Windowed Aggregation

Windowed aggregation converts a continuous sequence of events into bounded summaries. Instead of calculating a customer's total purchases forever, a pipeline might calculate purchases during each five-minute interval or spending over the preceding hour. Each result usually includes a grouping key, window boundaries, and aggregate values. For machine learning, these summaries turn raw clicks, transactions, or sensor readings into features such as requests per device, average transaction amount, or maximum temperature.

The window definition determines which events contribute to each result. Tumbling windows are fixed-length, nonoverlapping intervals, such as successive five-minute periods. Sliding windows overlap: a ten-minute window evaluated every minute lets an event contribute to multiple results. Session windows group activity separated by less than a configured inactivity gap, making them useful for user sessions with variable durations. Choose the definition according to the behavior the feature should represent, not simply the frequency at which a processing job runs.

Time semantics are equally important. Event time is when an activity occurred, while processing time is when the processing system handles it. A transaction generated at 10:02 but received at 10:07 belongs to the earlier interval under event-time processing. Processing-time windows are simpler but can change results when networks delay events or a stream is replayed. Normalize timestamps and define consistent interval boundaries, commonly including the start and excluding the end, to avoid ambiguous assignments.

Event-time engines commonly use watermarks to track progress and manage state for out-of-order events. A watermark is not proof that every earlier event has arrived. Configure lateness handling and decide whether delayed records should revise an existing result, be discarded, or be routed for correction. Emitting provisional updates provides fresher features, while waiting longer generally includes more delayed data at the cost of latency and retained state. Exact behavior depends on the processing engine and output mode.

In a cloud pipeline, a streaming processor reads events, parses and validates fields, groups records by an entity key, and maintains aggregate state for each active window. For example, Spark Structured Streaming can group transactions by account and a five-minute event-time window, then calculate count and sum. A mean should be maintained using sum and count rather than by averaging partial averages without weighting. Results can be written to an analytical store or a feature-serving database, with the entity and window boundaries forming a stable output identifier.

Windowing is not the same as batching. A Spark micro-batch or Lambda invocation can contain records belonging to several analytical windows; conversely, one analytical window can require state across many invocations. Lambda supports tumbling-window processing for supported stream event sources, but custom handlers must still manage their aggregate state and output behavior. Merely increasing a batch size or batching interval does not implement a correct business-time aggregation.

Operational trade-offs include state size, hot keys, processing latency, and recovery behavior. Longer windows, overlapping windows, high key cardinality, and greater lateness tolerance can increase resource requirements. Checkpointing supports recovery, but duplicate-free output also depends on source replay and sink write semantics; use idempotent writes or supported transactional integration. For ML, preserve consistent feature definitions between training and inference, and reconstruct only information available at prediction time. Including later events or corrections that were unavailable then can introduce leakage into historical training data.