Event-Time Processing

Event-time processing uses the timestamp describing when an event occurred to determine its place in a calculation. This differs from processing time, which is when an application handles the record, and ingestion time, which is when an ingestion system receives it. A sensor reading generated at 10:02 but delivered at 10:08 belongs to the 10:02 event-time interval, even though the processor sees it six minutes later.

This distinction matters when transforming streaming data into machine learning features. Suppose a fraud detection pipeline computes each customer's transaction count and average purchase amount in five-minute windows. Using processing time can place delayed transactions in the wrong interval and make features depend on network delays or consumer backlogs. Event-time windows preserve the intended relationship between transactions and the periods they describe.

A practical pipeline first parses and validates an event timestamp, normalizes its time zone and units, and retains a stable event identifier. It then groups records by an entity key, such as customer ID, and applies tumbling, sliding, or session windows. Transformations can include filtering invalid values, normalizing measurements, enriching records, and calculating counts, sums, or other features. Invalid timestamps and implausibly future-dated events need explicit handling because they can distort windows and progress tracking.

Out-of-order arrival requires the processor to retain state while waiting for additional records. A watermark represents progress through event time and supports decisions about window completion and state cleanup; it is not proof that earlier events can never arrive. Watermark behavior differs among engines. Spark Structured Streaming can use an event-time column with withWatermark and window aggregations, while Apache Flink supports timestamp assignment, watermark strategies, and configurable handling of late events.

For example, a pipeline might calculate five-minute transaction features while tolerating ten minutes of out-of-order data. A transaction from 10:02 arriving at 10:08 can still contribute while the relevant window state remains available. A much later transaction may be discarded, routed for reconciliation where supported, or incorporated through a separate correction workflow. The exact behavior depends on the engine, operator, watermark progress, and output configuration, not simply on elapsed wall-clock time.

Longer lateness tolerance can improve completeness but increases retained state, recovery work, and the delay before results become final. Early results can reduce serving latency, but downstream storage must support updates or versioned outputs if results change. Checkpoints help recover processing state, while stable identifiers and idempotent writes address duplicates and retries. Event-time processing alone does not guarantee exactly-once results or globally sort the stream.

In cloud implementations, managed stream processors or Spark jobs handle stateful event-time aggregations, while functions can validate, enrich, and transform incoming records. A function invocation or batch boundary is not automatically an event-time window. For machine learning, preserve timestamp semantics across training and inference, and distinguish occurrence time from availability time: historical features must not include information that was unavailable when a prediction would have been made. Monitor timestamp quality, watermark progress, late-record rates, state size, and feature freshness to balance correctness with operational cost.