What a hosting provider did today
Disk-full cleanups that delete error logs destroy the incident timeline.
Mid-incident, disk alarms fire. Someone deletes “old” logs — including the error log and slow log that held the only timeline.
Lost
- Crash fingerprints
- Alignment between app errors and DB waits
- Evidence of storage-layer glitches
Controls
- Ship logs off-box continuously
- Treat log retention as part of backup policy
- Document sacred paths in provider/customer runbooks
- Prefer environments where roles during incidents are explicit
Disk-full is an emergency. Deleting the crime scene is not a resolution.