Model Cascading

Model cascading organizes inference into stages rather than sending every request to the same expensive model. A common design starts with a small, inexpensive model that handles routine cases and invokes a larger or more specialized model only when necessary. Other cascades progressively filter or refine candidates, such as using a lightweight classifier to identify images that require detailed analysis. The defining characteristic is staged processing, with later models receiving selected requests or intermediate results from earlier stages.

For generative AI, an application might use a smaller pretrained foundation model to classify support tickets, extract fields, or answer straightforward questions from supplied documentation. The application checks whether the result meets acceptance criteria, such as a valid output schema, supported citations, or agreement with business rules. If the result fails those checks, it invokes a more capable model through an API. Using a small model for a bounded task does not inherently require fine-tuning; evaluate its pretrained performance before adding training complexity.

Escalation logic is critical. A language model's statement that it is confident is not a calibrated probability of correctness. Use validation results, calibrated classifier scores where available, evidence checks, or a separately evaluated quality estimator. Thresholds should reflect the business consequences of errors. For example, a document classifier may automatically accept high-confidence routine categories while escalating ambiguous documents. A high-risk request may bypass the inexpensive stage entirely, although a larger model still does not guarantee a safe or correct answer.

Cloud implementations typically place orchestration between the client and model endpoints. A function, application service, or workflow calls the first model, evaluates its response, and conditionally calls the next endpoint. Models can be accessed through managed foundation-model APIs or deployed on separately sized inference infrastructure. Independent endpoints allow each stage to scale according to its traffic and hardware needs. Pass the original request and necessary evidence to later stages rather than relying exclusively on an earlier model's potentially incorrect summary.

Evaluate the whole cascade, not just individual model benchmarks. If the first stage costs C1 per request, the second costs C2 per invocation, and fraction p of requests escalate, expected model cost is approximately C1 + p × C2, excluding orchestration and validation. This is cheaper than using the second model alone only when the first-stage and supporting costs are outweighed by avoided second-stage calls. Escalated requests also incur both stages' latency, so lower average cost can coexist with worse tail latency. Provision downstream capacity for escalation spikes, not merely the average escalation rate.

Cascading is related to, but different from, parallel ensembles that combine predictions through voting or averaging. It is also different from direct routing, which selects a model before task inference, and from an outage fallback, which switches models because an endpoint fails. A sequential pipeline that always invokes every model may support staged processing, but it does not provide the selective-execution savings of a conditional cascade. Early filtering errors can be irreversible: a relevant document discarded by the first stage cannot be recovered by a later model that never sees it.

Before deployment, measure end-to-end task quality, cost per successful request, escalation rate, and latency percentiles on representative data. Include difficult and high-risk cases, and examine false acceptance as well as unnecessary escalation. In production, version the models, prompts, and thresholds; monitor drift; bound retries and escalation depth; and enforce access controls at every endpoint. A cascade improves results only when its combined behavior meets the workload's quality, cost, and operational requirements.