Log Rotation Deleted Our Only Evidence of a Production Bug

Last month we had a weird intermittent error — an API endpoint returning 500s about twice a day, seemingly random. Users reported it, we acknowledged it, opened a ticket. Standard stuff.

When the engineer got to it three days later, the logs were gone. Our log rotation policy kept 48 hours of application logs. The error happened on Monday, the engineer looked on Thursday. The evidence had been rotated out and compressed into an archive that nobody knew how to quickly search.

The real problem wasn't the rotation policy

Keeping 48 hours of logs is fine for most things. The actual failure was that nobody set up alerting on 500 errors. We had uptime monitoring (is the server responding?) but not error rate monitoring (is the server responding correctly?).

The endpoint returned 500 twice a day out of maybe 10,000 requests. A 0.02% error rate doesn't show up in uptime dashboards. Response time was normal because the 500 was fast — the error happened early in the request lifecycle.

What we fixed

Three changes, in order of how much they actually mattered:

Error rate alerting. Any endpoint exceeding 0.1% error rate over a 15-minute window triggers a notification. This catches the slow bleed of intermittent failures before they become a pattern that users report.

Structured logging to a separate store. Application errors now go to a dedicated error tracking service in addition to local logs. The local logs can rotate on whatever schedule makes sense for disk space. The error store keeps 30 days of searchable error data.

Longer retention for error-level logs. We split the log rotation — INFO and DEBUG rotate after 48 hours, ERROR and above keep for 14 days. Barely costs any extra disk because errors are a tiny fraction of total log volume.

The intermittent 500? Turned out to be a race condition in a cache invalidation flow. Two concurrent requests could hit a window where the cache key existed but the underlying data had been deleted. Took 20 minutes to fix once we could actually read the stack trace.