Batch Processing

Batch processing operates on a bounded set of input data with a defined completion point. A batch might contain yesterday's transactions, a collection of uploaded files, or records added since the last successful run. Jobs can start on a schedule, when enough data accumulates, when an upstream dependency completes, or through an explicit request. A fixed schedule is common, but it is not required.

In cloud architectures, a typical pipeline lands source data in durable object storage, validates the inputs, transforms records using temporary compute resources, and writes results to an analytical store or another storage location. Separating storage from compute allows data to remain available between runs while processing capacity scales up for a job and scales down afterward. For example, a retailer might upload store sales throughout the day and process the completed daily dataset overnight to calculate inventory changes and revenue totals.

Ingestion frequency should follow the business freshness requirement. Daily ingestion may suit historical reporting, while hourly ingestion may support operational dashboards. An hourly schedule does not guarantee that data is available within one hour: collection delays, time until the next run, queueing, processing, and publication all contribute to end-to-end latency. When a requirement demands a reaction within seconds of an event, streaming or event-driven per-record processing is usually a better fit. Short, repeated micro-batches occupy a middle ground but still introduce buffering and execution delays.

Ingestion and transformation are separate decisions. Data can arrive continuously in storage while transformations run nightly, or a scheduled extraction can deliver a batch that is processed immediately. Exam scenarios often distinguish these choices through phrases such as daily reports, periodic uploads, large historical datasets, or near-real-time alerts. Choose the pattern from the required freshness and workload characteristics rather than assuming that all incoming data needs continuous processing.

Performance depends on more than adding workers. Partitioning inputs by date or another useful key enables independent work and reduces unnecessary reads. Columnar formats can improve analytical scans, while compressing data can reduce storage and transfer costs. Too many tiny files create scheduling and metadata overhead; poorly balanced partitions can leave most workers idle while one finishes an unusually large task. Benchmark representative input sizes and include resource startup time when estimating completion deadlines.

Reliable batch pipelines define exactly which records belong to each run. Use stable input manifests, extraction boundaries, or consistent snapshots where appropriate, and establish a policy for late-arriving records. Track completed inputs and make output writes idempotent so retries do not duplicate business results. Incremental jobs reduce repeated work, but their progress markers must advance only after successful publication. Retaining raw inputs supports backfills and correction of transformation errors.

Batch processing is not inherently cheap, slow, or limited to overnight execution. It can achieve high throughput through parallelism, but repeated full scans, idle resources, and oversized workers can waste money. Interruptible capacity can reduce compute costs when jobs tolerate retries or checkpoints. Monitor completion time, backlog, failed records, and data freshness, and ensure that sustained processing capacity can keep pace with incoming data.