The "Second Bill" Shock
It’s the new normal for scaling tech companies. You finally get your cloud spend under control, and then the other bill arrives. Datadog. New Relic. Splunk. And suddenly, you realize you are paying almost as much to watch your servers as you are to run them.
This happens because of a culture of “Just In Case.” Engineers are terrified of missing a bug, so they log everything. Every HTTP 200 OK. Every heartbeat. Every health check.
You are drowning in data, yet starving for insights.
The "Write-Only Data" Problem
We audited a client recently who was ingesting 5TB of logs per day. We asked: “What percentage of these logs are actually queried by a human or an alert?” The answer: Less than 2%.
The vast majority of your Observability spend is on “Write-Only Data”—logs that are generated, ingested, indexed, stored, and then deleted 30 days later without ever being seen by a human eye. You aren’t buying observability; you are buying digital storage for trash.
The Magic of Sampling
ou do not need to log every single successful request. If your API handles 1 million requests and 99.9% are successful, seeing 999,000 lines of “Status: 200 OK” tells you nothing new.
The Fix: Implement Dynamic Sampling.
Errors (5xx/4xx): Keep 100%. (High Value).
Success (2xx): Keep 1%. (Low Value).
This simple rule reduces your ingestion volume by ~90% while retaining 100% of the debugging context you actually need.
Are you a Digital Hoarder? Do you ingest logs that nobody reads? Take our 10-point audit to find your waste.
Treating logs like "free text" is a financial error. 💸In the cloud, every line of text has a price tag. If a log doesn't help you fix a bug, it is debt.🗑️Read how to lower your Observability Tax + Check your Score #Observability #DevOps #Datadog #Splunk #CloudCost #FinOps
The Cardinality Killer (Metrics)
While logs cost money to store, Metrics cost money to index. The #1 killer here is “High Cardinality.” This happens when an engineer tags a metric with a unique ID.
Good Tag:
env:production(Low Cardinality – 1 value).Bad Tag:
user_id:12345(High Cardinality – Millions of values).
Every unique tag combination creates a new custom metric. If you have 1 million users, you just created 1 million custom metrics. Your bill will explode immediately.
Conclusion: Log Smart, Not Hard Observability is insurance. But you don’t need to insure every single brick in the building separately. Focus on Signals (Errors, Latency, Saturation). Ignore the Noise.
Stop the Log Hoarding Audit your Ingestion, Retention, and Cardinality strategy.
Understanding that you are drowning in “Write-Only Data” is step one. Step two is identifying exactly which services are the biggest offenders.
We use a proprietary Observability Cost Framework at GYSP to help enterprises identify high-cardinality spikes, implement sampling, and stop paying for digital trash.
Stop guessing about your ingestion costs. Use the exact diagnostic tool we use with our enterprise clients to measure your log strategy.
👇 Take the Observability Cost Assessment Below:
