Log Retention

Log retention defines how long log records must remain available and under what conditions they can be deleted. It is separate from log collection: enabling a log source does not guarantee that its records will be stored for the required duration. A complete design specifies the sources, collection scope, storage destinations, retention periods, retrieval expectations, and controls that preserve the records.

Start with workload and security requirements rather than choosing one duration for every log. Administrative API activity, authentication events, network flows, application errors, database audit events, and web access logs answer different investigative questions. Determine which events are needed to detect threats, reconstruct an incident, satisfy contractual obligations, or troubleshoot failures. Confirm whether each source must be explicitly enabled, whether collection covers all relevant accounts and Regions, and whether filtering or sampling could remove required evidence.

Retention periods should reflect the time between an event occurring and someone needing to investigate it. A workload with delayed fraud discovery may need substantially more history than a short-lived development environment. Define both the total retention period and the period during which logs must be immediately searchable. For example, an organization might require 30 days of interactive investigation data and one year of durable archived records. These are example requirements, not universal compliance defaults.

Cloud implementations commonly separate a searchable store from a lower-cost archive. Recent logs support dashboards, alerting, and incident queries, while older records move to storage with lower ongoing costs. The trade-offs include indexing expense, query charges, archive minimum storage durations, retrieval charges, and restore delays. A storage class that requires hours to restore data is inappropriate when investigators must search the full retained history within minutes. Verify that the archive preserves the fields, timestamps, and context needed to reconstruct events.

Retention must work across the entire delivery path. Buffer expiration, failed delivery, missing permissions, or encryption-key problems can create gaps before data reaches its final destination. Monitor ingestion failures, unexpected drops in volume, and changes to retention settings. Test retrieval of older records and retain the necessary decryption keys for as long as the encrypted logs must remain readable. Check whether each system calculates expiration from event time, ingestion time, or object creation time; late-arriving records can otherwise receive less usable retention than intended.

Retention is not the same as immutability or backup. A configured expiration period does not necessarily prevent a privileged user from deleting records early. Where evidence preservation requires stronger protection, combine access separation with immutable storage or legal holds. In versioned object storage, lifecycle rules must also address noncurrent versions when actual removal is required. Legal holds may suspend normal expiration, while privacy requirements may limit unnecessary retention of personal information.

Do not assume that longer retention automatically improves monitoring. Alerts depend on timely ingestion, appropriate event coverage, and effective detection logic; archived records mainly extend retrospective analysis. Likewise, a service's built-in event history is not automatically a long-term archive. On AWS, CloudTrail event history provides the previous 90 days of management events in each Region; longer retention requires a separately configured destination or event data store with suitable retention settings.