Online inference is the use of a deployed machine learning model to process incoming application requests on demand. An application sends input to a model-serving endpoint and receives a prediction, classification, score, embedding, or generated response. Typical examples include scoring a payment before authorizing it, ranking products when a shopper opens a page, and answering a question in a chat application. The defining characteristic is that inference is triggered by current demand rather than a scheduled job over a collected dataset.
In the cloud, engineers commonly package a model and its inference code into a managed serving endpoint or a container-based application. A request is authenticated, validated, transformed into the model's expected input format, and passed to the model. The application then receives the processed output. Feature retrieval, network communication, preprocessing, model execution, and postprocessing all contribute to end-to-end latency. A fast model alone does not guarantee a fast application.
Real-time inference usually refers to synchronous online inference: the caller waits for an answer within the application's response-time budget. A checkout fraud check might need a result within a few hundred milliseconds, while a generative assistant may take longer and stream output as it becomes available. In machine learning discussions, real-time generally means sufficiently responsive for the application, not a hard guarantee that every request finishes before a strict deadline.
Batch inference processes a collected dataset as a job, such as scoring all customers overnight for a marketing campaign. It is appropriate when results are not needed immediately and can prioritize throughput and efficient resource utilization. Precomputing recommendations in a batch and retrieving them during a page request is not online model inference at retrieval time, although the application itself responds interactively.
Asynchronous inference accepts a request for later processing and returns the result separately, often through stored output, polling, or a notification. It is useful for long-running work, large inputs, or traffic that can tolerate queueing. An asynchronously submitted document-analysis request is demand-driven, but it does not have the same response contract as synchronous real-time inference. Exam questions commonly distinguish asynchronous, real-time, and batch inference by when work runs and how the caller receives results.
Serverless inference describes infrastructure management and capacity allocation rather than a separate timing requirement. A serverless endpoint can provide synchronous online inference without requiring the customer to manage individual servers. Depending on the service and configuration, scaling from zero or initializing a model can introduce cold-start delays. Provisioned capacity can improve predictability but adds cost, so serverless does not automatically mean lower latency or lower expense.
Production designs balance response time, throughput, availability, model quality, and cost. Continuously running instances can provide predictable readiness but incur charges during idle periods. Autoscaling helps handle variable traffic, although new capacity may take time to become ready. Engineers should load-test realistic payloads and concurrency, monitor tail latency and errors, and define timeouts and fallback behavior. They should also protect sensitive inputs, keep inference transformations consistent with training, and monitor prediction quality. Online inference does not mean online learning: generating a prediction normally does not update the model's parameters.
Amazon SageMaker