At-Least-Once Delivery

At-least-once delivery means that a messaging system can deliver the same accepted event multiple times while attempting to ensure delivery despite recoverable failures. The guarantee applies within a defined boundary, such as delivery from a broker to a consumer, and depends on the system's retention, retry, and failure-handling rules. It does not promise unlimited retries, permanent storage, or successful completion of downstream business operations.

The mechanism usually combines durable storage with acknowledgments or checkpoints. A producer submits an event, the transport retains it, and a consumer acknowledges it after completing the required work. If an acknowledgment is missing, the system can deliver the event again. For example, a log processor might successfully write a record to a search index and then crash before acknowledging the message. Redelivery is necessary because the transport cannot determine whether the write succeeded.

Duplicates can also originate at the producer. If a log shipper sends a batch but loses the response, it cannot distinguish a rejected request from an accepted request whose acknowledgment was lost. Retrying protects against missing logs but may insert the same events again. Batch APIs add another consideration: callers should inspect individual record results and retry failed records rather than automatically resubmitting every successful record.

In cloud log pipelines, subscriptions, streams, queues, and processing functions often introduce separate delivery boundaries. A log collection service might forward events into a stream, and a function might transform those events before writing them to storage. Reliability must be evaluated at every boundary. A downstream at-least-once transport cannot recover a log event that an upstream agent discarded before durable acceptance.

Consumers should therefore make repeated processing safe. Assign a stable event identifier before retries occur and preserve it throughout the pipeline. A search-index consumer can use that identifier as the document key so repeated writes replace the same document rather than create new documents. A metric aggregator needs stronger protection: blindly incrementing a counter for every delivery overcounts events. It can use a transactional deduplication record together with the counter update, or another design that makes the combined operation idempotent.

Deduplication itself has trade-offs. Its storage and lookup costs grow with event volume and the time window during which duplicates can arrive. Expiring deduplication entries too early allows old replays to affect results again. Hashing only the log text is usually insufficient because distinct events can legitimately contain identical messages. Separately recording an event as processed before committing its output can also cause loss if the consumer crashes between those actions.

Operational controls are equally important. Monitor ingestion failures, throttling, backlog age, consumer lag, retry volume, and dead-letter queues where supported. Configure sufficient retention and processing capacity to survive expected outages, and provide a replay procedure for failed records. Retries increase latency, request costs, and downstream load; backoff and capacity management help prevent a recovering destination from being overwhelmed.

For architecture and exam questions, distinguish delivery from processing and ordering. At-least-once delivery permits duplicates but does not inherently guarantee order or exactly-once effects. Real-time ingestion also does not mean instantaneous ingestion: retries and buffering can delay events. When missing logs is less acceptable than repeated logs, choose a reliable buffered path and pair it with duplicate-safe consumers rather than assuming the transport eliminates duplicates.