AWS Certified Generative AI Developer - Professional (AIP-C01) Course

Instruction hierarchy resolves conflicting directions by authority, not by position, length, or emphasis. Before implementing it, define a model-interface contract: the exact model and API, supported instruction fields and message roles, documented precedence, and handling of conversation history. Do not assume that a separate developer role exists everywhere. Amazon Bedrock Converse separates a system field from user and assistant messages, but system-prompt support depends on the model.

A responsible-AI policy becomes enforceable when each rule names its scope, detector, action, and failure behavior. “Protect customer information” is incomplete. Specify which information, whether it is forbidden before inference or only before disclosure, whether to mask or reject it, and what happens when detection cannot complete.

Instruction hierarchy answers: “Which instruction should govern when directions conflict?” Authority comes from the model interface’s supported roles and precedence rules, not from an instruction’s position, urgency, or wording. System-level instructions commonly outrank user requests, but exact roles vary by model and API. Compatible lower-priority requests still apply.

A prompt template constructs a model request from application-controlled instructions and runtime values. Keep the task, evidence rules, and response contract in the instruction locations supported by the chosen model interface. With structured messages, use actual role fields rather than strings such as SYSTEM: inside document text. Role support and instruction-following behavior vary by model.

Capacity enforcement starts with the selected model version and inference API, not a universal token limit. Record its shared context limit, any separate input limit, its output limit, and how it accounts for tool use and reasoning tokens where applicable. Revalidate these rules when changing models or APIs.

Conversation memory is application state that survives model invocations. The model uses the context supplied for an invocation; storing a transcript does not make that transcript available automatically, and supplying history does not train the model. A durable record and the context selected from it are different things.

Store one item per message, rather than repeatedly replacing a transcript list. Use a session-scoped partition key such as TENANT#t7#SESSION#s9. This supports a Query for one conversation without reading other sessions. Derive the tenant scope from authenticated identity; a key naming convention alone does not enforce authorization.

Custom text classification learns your categories from labeled examples; it does not discover your application's policy automatically. Define labels such as RESET_PASSWORD, CHECK_ORDER, and UNSAFE_REQUEST with inclusion rules, exclusions, and borderline examples. An unsafe-input label describes your policy, not a universal property of text. Include realistic benign requests, confusing near-matches, and adversarial wording—not just obvious unsafe examples.

Intent classification across turns assigns a goal using the latest message and relevant conversation state. After “Where is order A17?”, “Cancel it instead” changes the goal from check_order_status to cancel_order. The order reference can remain relevant; the previous intent must not become a permanent constraint.

Enable S3 Versioning on a general purpose bucket before publishing templates. Each subsequent write to a key receives a version ID; replacing the template creates another version instead of overwriting the previous version. A read without a version ID selects the current version, not a reviewed release.

A prompt regression is a decline against a defined quality criterion, not merely a different answer. To investigate it, identify the last acceptable release and the first suspect release. Preserve their exact prompt revisions and connect each to its application release and evaluation evidence. A movable label such as latest is not enough to identify what ran.

A paired comparison runs a baseline template and a candidate template against the same versioned test cases. Freeze the variable values, retrieved context, conversation history, tool results, and expected outcomes. Hold the model identifier or version, inference settings, output constraints, and request-building code constant. If the renderer itself is under investigation, compare renderer versions separately rather than changing both renderer and template.

An AI audit trail must connect who requested an action, what the application attempted, which model was used, which policies were evaluated, and what the user actually received. Record structured events with a schema version, event ID, event type, event time, actor, action, target, source context, and outcome. A readable message alone is not a reliable evidence contract.

Amazon CloudWatch supplies operational evidence through metrics and logs. AWS services publish service telemetry, while application instrumentation supplies details such as interaction identifiers, selected prompt versions, retry attempts, evaluation results, and business outcomes. CloudWatch does not automatically discover these application meanings: a quality score or approval decision must be produced and recorded by the application or another evaluator.

CloudTrail coverage is an event-selection decision, not simply a switch marked “Bedrock logging.” Supported Prompt Management changes, such as CreatePrompt, UpdatePrompt, and CreatePromptVersion, are management activity. Supported model invocation activity is recorded as data events. Recording management events alone does not establish an invocation history.

A decision record explains an application outcome without requiring the full interaction payload. Emit a structured JSON event containing a schema version, event time, decision ID, correlation ID, model identifier, prompt-template version, policy version, outcome, and reason codes. Use approved source references and pseudonymous subject identifiers where needed; these identifiers can still be sensitive.

Start with a stable event contract. In this lesson, the application emits exactly one attempt_complete event per completed model-call attempt, with numeric failed set to 0 or 1 and numeric latencyMs. An interactionId identifies the logical interaction; an attemptId identifies one attempt, including retries. Record the actual promptVersion, model, and task used by that attempt—not the version currently deployed.

A protected evidence archive must preserve both records and the ability to explain their provenance. This architecture follows CloudTrail evidence from an organization trail to an isolated archive and a controlled validation session. Arrows show evidence handoffs, not every authorization or public-key lookup. Application interaction records need their own collection path; CloudTrail is not a transcript of model prompts and responses.

Schema validation checks a value against a declared contract: required fields, types, permitted values, bounds, and other supported constraints. Successful JSON parsing establishes syntax, not schema conformance. A parsed object can still omit required fields or contain the wrong types.

A response-consumption boundary is the point before generated content is displayed, executed, stored for downstream use, or passed to another component. Treat the candidate response as untrusted there. A stronger system prompt is not enforcement: the application must decide whether the candidate may cross the boundary.

Semantic acceptance criteria specify what a response must mean, not which words it must use. For each validation case, record the user task, authoritative evidence and its version, required facts, prohibited inferences, and expected behavior when evidence is incomplete. A reference answer is one acceptable expression of these requirements, not necessarily the only correct wording.

Regression testing asks: did a change damage accepted behavior? A candidate is checked against established requirements and, where useful, a baseline version on comparable cases. For a foundation model application, accepted behavior might include grounded answers, valid tool arguments, and appropriate refusal. The reference is an approved standard, not merely whichever version scores higher.

Regression testing asks whether a change damaged behavior that previously worked. For GenAI, the baseline is not just a saved answer: it is an approved application configuration evaluated against a versioned test contract. Each case needs a stable identifier, input, relevant context, expected properties, evaluation criteria, and a category or criticality label. Different wording can still satisfy the same contract.

Context refinement means improving the evidence available for a particular answer, not simply adding more text. Start by identifying the claims the answer must support. For a return decision, those might be the applicable policy, the purchase date, the item condition, and any exception. A passage is useful when it supports one of those claims or resolves uncertainty about it.

Structured output is an interface contract, not just neatly formatted text. Start with what the consumer can accept: field names, types, required properties, allowed values, nesting, and treatment of unexpected properties. In JSON Schema, declaring a property does not make it required; required controls presence. Likewise, a required field can allow null: presence and availability of a value are different decisions.

Chain-of-thought prompting structures a problem through intermediate results instead of asking only for a conclusion. For application design, the useful target is a verifiable reasoning scaffold: a small set of public artifacts that another person or program can check. These might be extracted facts with source references, calculations with units, or a requirement-by-requirement assessment.

Controlled refinement starts with a behavioral contract, not a preferred wording. Define what counts as correct, which failures block release, and what evidence will demonstrate improvement. For an incident summarizer, this could mean preserving reported impact, separating observations from hypotheses, and never presenting an unsupported root cause as established.

A useful feedback record identifies which response was judged and under what conditions. Link the rating to a response identifier, timestamp, task, prompt version, model identifier or version where available, generation settings, and relevant retrieval or tool context. Preserve the necessary input and output securely, or retain controlled references to them. A prompt version alone cannot reconstruct a response if its evidence sources have changed.

Iterative prompt refinement repeatedly changes and tests instructions, examples, or output requirements to improve model responses. Its immediate object is the prompt. A developer can refine a prompt using a small collection of test inputs without establishing who handles production complaints or whether affected users receive a resolution.

Conditional branching determines which path executes. Sequential prompt chaining determines how separate prompt stages execute in dependency order: an earlier stage produces information that a later stage consumes. A branch selects work; a chain carries out dependent work.

Prompt chaining connects model interactions through intermediate outputs. The application defines transformations such as extracting facts, drafting an answer, and reviewing the draft. Passing a result onward does not itself delegate responsibility for achieving a goal.

A reusable stage template defines an input contract and an output contract, not just repeated wording. The input contract names the variables the stage accepts, their types, meanings, required status, and size limits. The output contract describes the result the next stage can consume: its structure, field meanings, allowed values, and how to represent missing information. Keep these requirements with the template as a versioned specification; placeholders alone do not express them.

A model response is a proposed decision, not an executable routing rule. In this Bedrock Flows routing segment, a prompt node produces a response and a Lambda function validates it before a condition node sees it. Asking for JSON does not replace parsing, required-field checks, type checks, and allowed-value checks.

A reusable prompt stage has a function-like boundary: named inputs enter, a model produces a candidate, and application code decides whether that candidate satisfies an intermediate contract. Define required fields, types, permitted values, schema version and failure behavior. Reuse comes from a stable interface, not from copying a conversation transcript.

A public leaderboard is evidence about its benchmark, not proof of workload fit. Start with a workload contract: required input and output types, successful behavior, unacceptable failures, and latency, cost, and governance limits. Check hard compatibility constraints such as modality support, licensing, and permitted data processing before benchmarking candidates.

A synchronous boundary keeps the caller waiting for inference on the same request. Budget for admission, network transit, queueing inside the serving system, generation, and response delivery. Use measured tail latency, not average model time, against the shortest applicable client or intermediary deadline. Streaming can improve time to first token, but does not eliminate generation time or connection limits.

Amazon SageMaker AI hosting deploys a model-serving workload that you configure. For a deployment using your own model artifacts, you control the supplied weights, tokenizer, configuration, and any adapters, subject to their licenses. You also select the serving image. AWS manages the hosting service; it does not take over responsibility for whether those artifacts work together. These hosting capabilities belong to SageMaker AI, within the broader Amazon SageMaker platform.

A SageMaker AI deployment connects three resources. CreateModel defines the serving image, model artifacts, execution role, and container environment. CreateEndpointConfig references that model through production variants and specifies hosting capacity. CreateEndpoint creates the addressable endpoint from that configuration. A model resource alone does not run inference.

A synchronous inference contract needs an end-to-end deadline, a target latency percentile, and an acceptable rate of failed or late responses. Network travel, admission, feature reads, transformations, queueing, inference, and response handling all consume that deadline. Measure complete requests: adding each stage's p99 does not calculate the request's p99.

Amazon OpenSearch Service provides the search backend: it indexes vectors and associated fields, executes search requests, and returns matching records. Managing the search infrastructure does not, by itself, define how source documents become useful grounding context.

A useful asynchronous boundary publishes a committed business fact, not a request to run a particular model. OrderChanged says what happened; SummarizeOrderWithModelX couples the producer to a consumer's implementation. The legacy transaction should complete without waiting for inference. Its success means the business change is durable, not that every subscriber has finished.

AWS Lambda executes application code. An ordinary event-function invocation owns its local call stack while it runs; successful completion does not create a durable record of an application's remaining steps. Calling several services inside one handler makes that handler responsible for their sequencing.

A Choice state selects a transition; it does not invoke a model or perform moderation itself. Tasks produce structured facts such as a moderation verdict, request category, or completion flag. Choice rules interpret those facts. With QueryLanguage set to JSONata, each rule has a Boolean Condition and a fixed Next state name. For example, {"Condition":"{% $states.input.verdict = 'deny' %}","Next":"Reject"} routes an explicit denial to a rejection path.

A service accepting a request is not the same as finishing its work. With request-response, a Task advances when the integrated API call returns. For an asynchronous job-submission API, that response normally means submission succeeded, not that processing finished.

A Map state applies an ItemProcessor workflow to a collection. For query fan-out, an item can be one decomposed query or expansion. For prompt regression, it can be one case with a stable case ID, prompt version, input, and expected checks. Map schedules independent work; it does not itself rank retrieved documents or decide whether a prompt passed.

A failed Task first goes through its matching Retry policy. Step Functions scans retriers in order and uses the first matching one. Only when retries are unavailable or exhausted does it scan Catch handlers in order. Put specific errors before broad matches; States.ALL must appear alone and last in its array. Retry transient failures, not invalid requests or authorization failures.

Capability-based routing chooses which specialized foundation models should handle a request or its subtasks. Maintain a versioned capability registry containing supported modalities, task-specific evaluation results, input limits, permitted data use, and measured resource requirements. First exclude ineligible models; then choose among eligible models using the request's quality, latency, and cost constraints.

A golden set is a reviewed collection of reference cases, not a collection of polished model answers. Each case needs a stable identifier, task input, relevant context, evidence snapshot with source locations, and a scoring rubric. Record the evidence's effective date and scope: an answer can be correct for one product version but wrong for another.

A task-specific rubric converts “good output” into judgments that another evaluator can reproduce. Define each dimension, the evidence available to the judge, scoring anchors, and rules for insufficient evidence. For a support answer, score relevance to the request, factual accuracy against authoritative information, consistency with instructions and other claims, and fluency separately. Fluent wording must not compensate for an incorrect answer.

A controlled comparison starts with a decision contract: the supported workload, unacceptable failures, acceptance thresholds, and rule for choosing among acceptable candidates. Separate hard gates from preferences. A required quality floor or latency ceiling is a gate; lower cost among candidates that pass is a preference. A weighted average must not let excellent performance elsewhere compensate for a failed mandatory requirement.

A GenAI proof of concept supports an investment decision only when it estimates incremental impact: the difference from a credible alternative, not simply performance before and after introducing AI. Before collecting results, specify the business objective, baseline workflow, eligible task population, evaluation period, quality and safety thresholds, and measurement plan. The baseline may be an existing manual process or simpler software; it should include improvements that would happen without GenAI.

The AWS Well-Architected Framework supplies six perspectives for judging a workload; the Generative AI Lens adds GenAI-specific questions and best practices. Together they guide reusable standards, but neither prescribes one architecture for every deployment. The workload boundary includes data, model interactions, infrastructure, operating procedures, and accountable people.

The AWS Well-Architected Tool records workload reviews against architectural best practices. The Generative AI Lens adds questions and guidance for generative AI workloads. Define the workload boundary before answering: include the application, its model interactions, relevant data flows, and operational responsibilities. Reviewing only the model endpoint can leave retrieval, output handling, and downstream actions outside the assessment.

A reusable infrastructure module is a versioned architectural contract, not just a template with variables. Its contract includes accepted inputs, promised outputs, security properties, and update behavior. Start with workload reviews using the AWS Well-Architected Framework and its Generative AI Lens, then translate applicable decisions into explicit requirements. For this lesson's module library, those requirements include encrypted data stores, blocked public data access, scoped workload permissions, and operational audit records without prompt bodies.

A CloudFormation module packages a reusable template fragment as a registered type. Consumers declare the type in their templates, and CloudFormation expands it into resources managed by the consuming stack. This is not a separately managed nested stack. Start with a narrow contract: keep required resource settings fixed, and expose parameters only for supported variation.

A reusable construct is more than a shortcut for creating resources. It defines a contract: supported inputs, infrastructure guarantees, and capabilities consumers can use. A custom class extends Construct, accepts scope, id, and typed props, and calls super(scope, id). Create its child constructs with this as their scope so the pattern forms its own subtree.

A CDK construct library distributes executable modeling code. Your CDK application instantiates constructs, and synthesis turns the resulting construct tree into CloudFormation templates. L1 constructs closely expose CloudFormation resource schemas; L2 constructs add intent-oriented APIs and defaults; higher-level patterns compose multiple resources. A construct is not itself a deployed resource.

For an autoregressive language model, prefill processes the prompt and builds attention key/value state, usually called the KV cache. Many prompt positions can be processed in parallel, but more input still adds work and cache storage. Decoding then generates successive tokens using that state. Each new token depends on previous tokens, so generating a longer answer adds sequential work. Longer contexts can also increase decoding work; input length does not affect only prefill.

A provider-independent contract describes what the application requests, not a particular provider's JSON. Define messages, generation settings, required capabilities, and a versioned result containing content, completion status, usage, and diagnostic metadata. A logical model name identifies an approved configuration target rather than exposing a provider model identifier.

Runtime routing configuration separates a routing decision from its implementation. A versioned document can map a logical route such as summarize to an adapter, model identifier, region, inference settings, and an approved fallback. Record the configuration version separately from the application release so either can change independently.

The Strategy Pattern puts implemented inference strategies behind one contract. Here, that contract accepts messages, generation settings, and request requirements, and returns content, usage, and a normalized completion status. A deployed Python registry in AWS Lambda maps approved strategy identifiers to implementations; the caller never selects an arbitrary provider URL.

Adapter makes an existing implementation usable through the contract an application expects. It answers: “How do I call this provider without exposing its interface?” For example, an adapter maps application messages into a provider request and converts the provider response into an application result. Its defining responsibility is translation, not deciding which provider deserves the request.

AWS AppConfig delivers configuration independently of application code. A routing document can contain a schema version, a route name, a provider identifier, a model identifier, and inference parameters. Existing application logic reads those values and selects an already implemented provider adapter.

A stable API contract describes what clients send, not what one provider happens to accept. For example, expose POST /generate with a JSON body containing prompt and a logical route query parameter. Keep provider field names and model identifiers behind that contract. This lesson uses API Gateway REST APIs: their request models, validators, and VTL body templates are not the HTTP API feature set.

A shared circuit breaker lets failures observed by one workflow execution stop later calls from other executions. Closed admits calls; open blocks calls; half-open admits a controlled recovery probe. Step Functions Retry and Catch handle an execution's errors, but do not themselves share health information across executions.

Graceful degradation deliberately reduces what an application does while preserving what it must guarantee. Separate essential outcomes from optional features: a support assistant can replace generated advice with approved documentation and an escalation route. It must disclose that reduction rather than present a template as a generated, evidence-backed answer.

Failure-aware inference routing chooses an eligible destination before spending time on inference. A separate circuit breaker for each regional endpoint and model prevents repeated calls to a destination already judged unhealthy. Independent regional endpoints contain a regional inference failure only when their required models, permissions, capacity, and supporting dependencies are independently usable.

A full-weight export contains the complete tuned model weights, including every shard and its index when weights are sharded. Package the model configuration, tokenizer assets, special-token mappings, chat template when used, and required loading code alongside them or through pinned references. An inference export need not contain optimizer state; a resumable training checkpoint also needs training state such as optimizer, scheduler, and random-number-generator state.

A LoRA adapter supplies a learned update to selected base-model weights: W + sBA. It is not a complete model. With separate serving, a LoRA-capable inference runtime loads the frozen base and adapter separately and applies the update during inference. Sharing one base across adapters can reduce duplicated storage and memory, but adapter loading, caching, request routing, and isolation still need explicit support. Separate artifacts do not automatically provide safe hot swapping.

A Model Package Group holds numbered model package versions in SageMaker AI Model Registry. Register each customized release as a new version, with its inference specification, artifact locations, evaluation references, and identifying metadata. Registry records reference artifacts; they do not preserve the bytes behind a mutable S3 key or container tag.

AWS CodeBuild executes commands in a managed build environment. A build project defines the environment and other build settings; a buildspec supplies commands and output definitions. Compilation, regression tests, security scans, and packaging are work performed by those commands, not by CodePipeline itself.

A GenAI release is more than model weights. Its immutable manifest pins the model or model package, adapter and compatible base model, inference image digest, prompt templates, retrieval configuration, and evaluation policy. Promote those same references rather than rebuilding between environments. This architecture shows the release-control flow, not inference traffic or every artifact download.

A controlled release promotes a specific revision, not whatever source happens to be newest when deployment starts. Configure a V2 pipeline with Validate, Approve, and Production stages. Associate test results, security-scan evidence, and approval with that execution and its deployment artifact.

A foundation-model rollback unit is the complete serving release: exact base-model revision, fine-tuned weights or LoRA adapter, tokenizer, inference container, prompt template, generation settings, and request/response contract. An adapter must match both its base model and serving implementation. Preserve the combination that passed evaluation; restoring old weights inside a new runtime is not necessarily a rollback to known-good behavior.

A fine-tuned model is a release candidate, not a production release merely because training succeeded. Evaluate the exact serving combination against the incumbent: task quality, safety, retained general capabilities, latency, and resource use. For LoRA or adapters, exercise the actual base-model and adapter loading path. Define thresholds and critical data slices before running the evaluation; a better average must not conceal a safety regression.

A defensible quality baseline describes a known corpus under a recorded measurement procedure. Start with document-level measurements: extraction success, extracted character count, token count, language estimate, duplicate fingerprint, and required metadata presence. Preserve document identifiers and processing versions. An extraction failure must remain a measured outcome, not disappear because the profiler only receives successfully extracted text.

Data profiling describes what was observed: field types, missing-value frequencies, distinct counts, length distributions, or relationships between fields. Schema validation compares data with a declared structural contract: required fields, permitted types, nesting, and explicitly encoded constraints. Profiling produces evidence; validation produces a judgment relative to expectations.

A foundation model dataset needs a declared contract, not just parseable files or an inferred table schema. For a JSON prompt-response record, explicitly require record_id, prompt, and response; define their types, null handling, nested structure, and policy for unexpected properties. Pin the schema version and validator behavior for each run.

Amazon SageMaker Data Wrangler is the stronger fit when an ML practitioner needs to inspect data, compare transformations, and decide how to prepare it. Its interactive interface combines visual analysis, built-in transformations, and custom transformations. The central question is exploratory: which preparation choices preserve useful information and produce suitable model inputs?

A reusable Data Wrangler flow needs an explicit data contract: required fields, accepted types, allowed content, and rejection rules for the intended foundation model task. For example, require a stable document identifier and nonblank text, and define how to handle duplicates and sensitive content. A statistically unusual record is not necessarily unsuitable; suitability depends on this contract.

A DQDL ruleset declares checks over the dataset supplied to an evaluation. For example, Rules = [RowCount > 0, Completeness "text" >= 0.99, IsComplete "id"] checks that the batch is nonempty, at least 99% of text values are present, and every id is present. The evaluated scope is the input you supply—not automatically the entire source table.

CloudWatch observes measurements; it does not evaluate model outputs. Application code, human review, or an external evaluator must decide whether an answer is supported, assign quality scores, or label outcomes. CloudWatch stores the resulting custom metrics, aggregates them, applies metric math, and displays trends. A metric named FairnessScore has no automatically supplied fairness definition.

A CloudWatch metric alarm evaluates a selected statistic against a threshold and has three states: OK, ALARM, and INSUFFICIENT_DATA. CloudWatch evaluates measurements supplied by your application; it does not judge prompt quality itself. A regression alarm therefore needs a quality score produced by an evaluator, not simply a count of model invocations.

SageMaker Processing, part of SageMaker AI, runs finite data-preparation jobs on managed compute. For multimodal preparation, a custom container supplies your codecs, tokenizers, libraries, and transformation code. Processing does not infer how text, images, audio, and table rows belong together.

Amazon Textract blocks are a graph: each block has an Id and a BlockType, and relationships point to other blocks by ID. Concatenating every block's text loses structure and can repeat content because LINE blocks and their WORD children describe the same text at different levels. Use document analysis with the FORMS, TABLES, and LAYOUT features needed by your transformation; text detection alone does not provide those analyses.

For foundation-model ingestion, treat the structured JSON transcript as the source record, not just the readable transcript string. This lesson uses standard batch transcription. Configure the job according to how the recording represents speakers: speaker diarization distinguishes voices in shared audio, while channel identification preserves separate audio channels.

A fusion-ready artifact contains usable content plus enough metadata to establish what it describes and where it came from. Record an entity or case ID, modality, source object version or checksum, evidence location, preparation version, and quality status. Preserve originals: a normalized derivative is not a replacement for its source.

Serialization for inference starts with the target model and API operation, not with a JSON encoder. A request can be valid JSON yet fail because its field names, types, content blocks, or generation parameters do not match that interface. Treat the model identifier, operation, and documented request schema as one contract.

Named Entity Recognition (NER) detects mentions and assigns categories. In Maya joined Northwind Ltd. on March 12., a recognizer might return Maya → PERSON, Northwind Ltd. → ORGANIZATION, and March 12 → DATE. Its output describes spans in the input, not verified people, businesses, or events. Categories depend on the recognizer.

Text normalization reduces differences in representation, not differences in meaning. Define a field-specific normalization contract: permitted edits, protected content, required structure, and conditions that require abstention. Keep the original separately; a normalized copy is not evidence that the source was accurate.

Deduplication should remove redundant representations, not meaningful differences. Start with an explicit equality contract: which content transformations preserve meaning, and which boundaries prohibit consolidation? For example, normalize Unicode and line endings where appropriate, but preserve negation, quantities, exceptions, and effective dates. Case folding and punctuation removal are unsafe for some technical content. Keep the original alongside the normalized representation.

A trustworthy annotation makes two separate claims: where the mention occurs and which category fits its context. Preserve a source version, the exact text submitted, the returned mention text, start and end offsets, label, model identity, and score. Amazon Comprehend entity offsets identify positions in the submitted text, with an inclusive beginning and exclusive end. They are not byte offsets; keep the character-index convention explicit when moving results between runtimes.

Amazon Comprehend’s DetectEntities operation identifies entity mentions in text. Each returned entity includes Text, Type, Score, BeginOffset, and EndOffset. Supply the supported LanguageCode appropriate to the input; entity detection does not automatically choose it for you.

Consider How do I fix E1047 after rotating database credentials? The identifier E1047 names a specific failure; the remaining words describe an intent. In this lesson, semantic retrieval means retrieving by similarity between compatible query and passage embeddings. It can connect “rotating credentials” with “changing a password,” but similarity does not guarantee that different error codes remain distinguishable.

A vector index has a retrieval contract: the embedding model and version, preprocessing, vector dimension, and distance metric must agree between indexed documents and queries. Equal dimensions do not make embeddings from different models comparable. OpenSearch can accept correctly shaped vectors while their nearest neighbors are semantically meaningless.

RDS for PostgreSQL with pgvector stores embeddings beside relational metadata and executes similarity queries through PostgreSQL SQL. Amazon S3 retains the source documents, addressed by bucket, object key, and, when versioning is enabled, version ID. A filesystem path or block address is not that object identity.

A neural search pipeline turns text into vectors and retrieves semantically related passages. A hybrid query adds a keyword branch; a search pipeline then combines the branches into one ranking. The diagram shows both ingestion and retrieval through one OpenSearch domain, with a Lambda function acting as the client.

A multi-stage search pipeline spends inexpensive retrieval work broadly, then applies more expensive relevance inference to a bounded set. Candidate retrieval finds plausible passages; reranking compares the query with those passages and changes their order. A reranker cannot recover a passage it never receives.

For S3 general purpose buckets, system metadata includes service-controlled properties such as Last-Modified and user-controlled headers such as Content-Type. You cannot assign an arbitrary value to Last-Modified. It describes the stored object, not the document's publication date. Store a business timestamp explicitly instead.

A metadata contract defines names, types, required values, business meanings, and relationships—not just a JSON shape. Retrieval code depends on these promises. For this lesson, published_at means the source's publication event, while effective_at means when its guidance becomes applicable. Neither means when ingestion ran.

A metadata contract defines each field's meaning, type, authoritative source, allowed values, missing-value policy, and owner. For this lesson, effective_at means the business-effective instant, represented as integer UTC epoch seconds. It is not the S3 upload or LastModified time. Reject ambiguous dates unless the source contract supplies a timezone.

ANN tuning is a constrained optimization problem: reduce search cost while meeting recall, latency, memory, and freshness requirements. Start with representative queries and a fixed corpus snapshot. Keep the embedding model, preprocessing, distance metric, metadata filters, and requested k identical between exact and approximate retrieval. Confirm the baseline uses an exact search plan; an index being available does not prove which plan ran.

A vector shard owns a subset of documents plus the structures needed to search their vectors. In OpenSearch, primary shards define these partitions; replica shards are additional copies of them. A normal search selects one available copy of each participating logical shard, rather than searching every replica.

Distributed ANN has three distinct decisions: which domain indexes to search, which candidates each shard returns, and which results survive the global merge. The diagram follows one selected domain index with two shards. It shows logical roles, not a requirement for one node per role.

Hierarchical indexing makes progressively finer decisions before searching individual vectors. An upper level can hold domain descriptions or representative vectors; selected domains can lead to document groups and then passages. Each routing decision removes branches from further consideration. This lesson uses an application-managed hierarchy; an ANN algorithm’s internal navigation hierarchy is a different mechanism, not the same tuning control.

Hierarchical indexing organizes retrieval through relationships between levels. An upper-level result identifies a more specific search space: a domain points to its documents, for example, or a broad vector neighborhood leads toward finer neighborhoods. The relationship guides where retrieval goes next.

A primary shard holds a partition of an OpenSearch index. Adding primary shards can distribute vector storage and search work across nodes, but an unrestricted query generally searches one copy of every targeted primary shard. More shards therefore create more shard-level searches, candidate lists, and coordination work. Fan-out is the number of search targets contacted for one query, not simply the number of nodes available.

A versioned OpenAPI contract describes HTTP operations, parameters, request and response schemas, and security requirements. For this lesson, POST /v1/evidence/search has the stable operationId searchEvidence. Its request requires query and customerId, with an optional bounded limit. Its success data contains an ordered list of evidence items with id, text, and source.

A source connector must extract identity, content, provenance and permissions, not just download files. It authenticates with least privilege, follows pagination, handles throttling with backoff, and understands the source's change cursor. This reference design assumes a wiki API that exposes content changes, permission changes and deletion records.

Amazon AppFlow extracts supported SaaS objects through a connector and writes mapped records to S3. Before configuring a flow, verify the connector's source objects, authentication, supported triggers, incremental fields, and attachment behavior. Access to a record does not imply that its linked document content is exported.

A repository adapter translates document changes into a consumer's ingestion contract. Here, a repository publisher submits change references to a standard Amazon SQS queue; a Lambda adapter fetches the referenced version and submits it to an Amazon Bedrock Knowledge Bases custom data source. Queue acceptance means work recorded, not document searchable.

Change data capture (CDC) identifies changes in a source and makes them available to consumers. Its contract concerns source records: which record changed, which operation occurred, and what position or ordering information is available. Log reading, triggers, and suitably designed polling are capture methods; the available details depend on the method.

A custom application publishes facts through PutEvents, naming the custom bus and supplying Source, DetailType, and JSON Detail. EventBridge routes the resulting envelope using event patterns. Producers name the event contract, not its consumers.

Idempotency means repeating a logical update does not change its intended effect. A vector write can succeed even when its response is lost, so a retry must mean “finish this update,” not “create another set of chunks.” Repeated execution is allowed; duplicate effects are not.

A maintenance worker needs two different positions: the source position of a change and the committed progress of its derived updates. Reading a record does not prove that its vectors are current. Here, producers send complete document versions or explicit deletions to a retained stream, using the document ID as the partition key. An ordered source version protects against duplicated or stale producer records; arrival time alone does not establish source order.

A source change identifies a document, but retrieval operates on its chunks. Use a stable document key, including the tenant or source namespace when necessary. Every vector record should carry that document key and have a deterministic chunk ID scoped to it, such as tenant7/doc42/chunkerA/2.

A replay-safe vector projection needs a source version for each logical record: a revision or source position that orders its changes. Compare versions only within their defined scope, usually one record. Arrival time, embedding completion time, and an arbitrary event identifier do not establish source order.

A vector store is a derived view, not the authority. Define acceptance against a published source snapshot: document IDs, application-assigned revisions, content hashes, deletion tombstones, and access metadata. Use an immutable manifest to identify a consistent snapshot; listing changing objects is not itself a consistent source contract.

A fixed token window needs three values: window size W, overlap O, and stride S = W − O, with W > 0 and 0 ≤ O < W. For a token sequence of length N, start at zero and emit [start, min(start + W, N)). Stop when the window reaches N; otherwise advance by S. Empty documents emit no chunks. The final window can be shorter; do not emit another window after one already reaches the end.

A child chunk is a search unit; a parent chunk is an expansion unit. First identify bounded sections that preserve a useful explanation, procedure, or policy clause. Then split each parent into searchable passages. A runbook parent can contain prerequisites, execution, and recovery checks, while its children isolate those passages.

Semantic chunking aims to produce passages that remain meaningful when retrieved independently. In a common embedding-based implementation, the chunker splits extracted text into sentences, builds a comparison group around each sentence, embeds those groups, and compares neighboring vectors. These comparison groups are not necessarily the final chunks. Their vectors help locate candidate boundaries; the final passages need their own embeddings for retrieval.

Fixed-size chunking chooses boundaries primarily by a length target or limit, measured in characters, words, or tokens. Semantic chunking uses meaning to decide which text belongs together and where a change justifies a split. The distinction is the decision rule, not whether a resulting passage happens to read coherently.

Fixed-size chunking creates independently searchable windows based on a configured length. In the baseline retrieval path, each selected window is also a unit of context supplied to the model. Source metadata can identify its document and location, but does not itself cause surrounding passages to be returned.

Hierarchical chunking defines relationships between levels: a smaller passage belongs to a larger parent, and retrieval can use that relationship to recover broader context. Semantic chunking selects boundaries according to meaning. It tries to keep a coherent idea together and separate changes of topic. One describes how chunks relate; the other describes where text is split.

Compare embedding models on the retrieval task you actually operate, not on vector size or a public leaderboard alone. Build a representative benchmark from domain queries and a fixed corpus snapshot. Include terminology, abbreviations, exact identifiers, ambiguous requests, supported languages, and realistic query lengths. Weight results by expected traffic, but report important slices separately so a strong average cannot hide a critical language or domain failure.

Select an embedding configuration against a domain retrieval benchmark, not vector size or a general leaderboard. Collect representative queries with judged relevant passages, including terminology, abbreviations, languages, exact identifiers, and difficult near-matches. Keep a held-out evaluation set separate from configuration tuning. Freeze chunking, corpus versions, filters, and retrieval depth when comparing embedding candidates so those changes do not confound the result.

A worker batch and a model request are different units. A Lambda invocation can receive several queue messages and make several model calls. Do not assume that an embedding API accepts a list of texts: Titan Text Embeddings V2, for example, accepts one input text per invocation. Check the selected model's current request schema and limits.

A poor generated answer does not identify an embedding failure. Start with a controlled relevance set: representative queries, human-judged relevant passages, and plausible but irrelevant passages. Freeze the corpus snapshot, query text, relevance judgments, embedding configuration, and access-filter scope. Confirm that the relevant information actually exists in the eligible corpus; missing content is a coverage problem, not evidence against the embedding.

Embedding compatibility is a contract, not a dimension check. Query and document encoders must produce vectors in a shared space; different encoders can work when explicitly trained as a compatible pair. Preserve the model version, required query/document formatting, dimensions, and normalization settings. Use the similarity measure the model expects. Equal-length vectors from unrelated models are not meaningfully comparable, and similarity is not a probability of relevance.

pgvector adds vector types, distance operators, and similarity indexes to Aurora PostgreSQL; it does not generate embeddings. Check the extension versions available for your Aurora engine with SELECT name, version FROM pg_available_extension_versions WHERE name = 'vector';. Using a role authorized to install extensions, run CREATE EXTENSION IF NOT EXISTS vector; in each database that needs it. Verify the installed version with SELECT extversion FROM pg_extension WHERE extname = 'vector';. Available index features depend on that version.

Index health is a set of constraints, not a single score. Measure approximate-neighbor recall@k against exact top-k results for a representative query sample: the overlap divided by k, assuming at least k eligible records. Use the same corpus snapshot, filters, distance metric, and tie policy. This measures approximation loss, not whether the documents answer the user's question.

Amazon Kendra has been in maintenance since June 30, 2026. It is closed to new customers; existing customers can continue using it. Do not select it for new workloads. AWS's migration guide is https://docs.aws.amazon.com/kendra/latest/dg/kendra-availability-change.html. This lesson explains its retrieval responsibilities as AWS offered them.

Candidate fusion starts with two bounded result lists: the top N_L lexical hits and the top N_V vector hits. Both must represent the same searchable corpus and enforce the same authorization, tenant, version, and other mandatory constraints. Express these as one logical eligibility predicate, even if each retrieval path implements it differently. Use stable chunk identifiers so a hit returned by both paths becomes one candidate with two contributions.

Reranking scores an already retrieved set against the query, then selects its strongest evidence. Let N be the candidate count and K the returned result count, with K no greater than N. Reranking can promote useful passages buried in those N candidates; it cannot recover a passage absent from them.

Hybrid search combines complementary retrieval paths over the searchable corpus. Let L be the lexical results and V the vector results. Their union, U = L ∪ V, contains every distinct candidate returned by either path. Combining paths can introduce evidence absent from one path alone, but does not guarantee that the evidence is relevant.

Query decomposition separates the facts a question requires. Query expansion broadens the search expressions used to retrieve evidence for the same information need. Both can produce multiple searches, so query count does not distinguish them.

Vocabulary mismatch occurs when a request and relevant documents express the same information need differently. For example, a user asks about “recovering accidentally deleted files,” while documentation describes “restoring previous versions.” Query expansion supplements the original query with alternative search expressions. Query rewriting, in the narrower sense used here, replaces the formulation used for retrieval with a different formulation of the same need.

Query rewriting produces a retrieval representation of a request, not an answer or merely cleaner prose. Its invariant is the user's information need: entities, requested relationships, negations, dates, jurisdiction, and other explicit constraints must survive. Natural-language retrieval does not promise identical result sets, so preservation needs both validation and empirical testing.

Bounded query expansion searches the original query plus a small, validated set of alternative expressions. Preserve identifiers, negation, dates, and explicit constraints. For “Why does API X fail under load?”, throttling and timeouts are search hypotheses—not established diagnoses. Reject variants that invent a cause or change the requested scope.

A useful decomposition is a typed dependency graph, not a list of search phrases. Consider: “For tenant A’s Orion production database as of 2025-01-31, does configured backup retention meet policy?” First identify the database; then retrieve its applicable policy and configuration. The latter two branches can run independently once the database identity is known.

Model Context Protocol (MCP) and tool calling address different boundaries. MCP standardizes communication between an AI application and servers exposing external capabilities. Tool calling lets a model produce a structured proposal containing a tool name and arguments. That proposal does not specify how the application must reach the implementation.

A retrieval contract defines the meaning of a search request and its result independently of the interface carrying it. For example, search_passages_v1 accepts query text and typed filters, then returns ranked passages—not a generated answer. Keep the embedding profile and index configuration behind the contract: the service embeds text using a profile compatible with the indexed vectors.

A model-generated tool call is an untrusted proposal. The application resolves its name through an allowlist, parses its arguments, rejects unsupported fields, and checks types, ranges, and business rules. A schema-valid refund amount can still exceed the refundable balance. Never turn a tool name into an arbitrary function lookup or interpolate arguments into executable code.

An interoperable MCP tool service separates the host's client, the protocol endpoint, and backend operations. This example runs the host on Amazon ECS and a stateless MCP implementation on AWS Lambda, behind Amazon API Gateway. The diagram shows only request and backend-call directions; responses return along those paths.

AgentCore Gateway exposes configured retrieval operations as discoverable MCP tools. An agent uses the same MCP interface whether the implementation is a Lambda function or an HTTPS API described by a supported OpenAPI schema. Gateway adapts the protocol; it does not perform the retrieval itself.

A supervisor owns the overall task, delegates bounded subtasks, and decides how returned results affect the next action. A handoff transfers responsibility for continuing the task to another agent; it is not merely a request for a specialist result. Define the receiving role, permitted next actions, acceptance conditions, and what happens if the recipient refuses or fails. Otherwise, agents can repeatedly redirect work without an accountable owner.

Shared workflow state is the application’s authoritative record of a task: accepted results, outstanding work, approvals, and the current version. Give it an explicit scope, such as tenant_id + task_id. A result becomes shared state only through the application’s acceptance and commit rules—not because an agent said it was finished.

Multi-agent orchestration defines a responsibility contract: which agent owns a task, what it must deliver, and how that contribution fits the application's shared goal. Agents can use the same underlying model yet have different responsibilities. Their coordination comes from those assigned responsibilities and relationships, not from model identity or connectivity.

A correlated trace connects one task's delegation, agent execution, handoffs, and tool calls. Give each run a stable application identifier and propagate trace context across execution boundaries. A span identifies one operation; parent relationships describe nested work, while span links can represent asynchronous or multiple causal predecessors. A shared run identifier alone helps searching but does not establish those relationships.

A production ReAct loop alternates a model decision with an actual observation. In this implementation, AWS Step Functions owns the state and transitions; the model proposes the next action but cannot execute it directly. The diagram shows service interactions, not individual state-machine states.

Chain-of-thought prompting encourages intermediate reasoning through instructions or worked examples. It can help organize dependent calculations and comparisons using the information available to the model. Its defining feature is reasoning structure, not access to external information: asking a model to explain a lookup does not execute that lookup.

An executable plan is a dependency graph, not merely a numbered list. Give each action an identifier, an allowed tool, typed arguments, prerequisites, expected outputs and a completion check. Keep verified facts separate from assumptions that still require evidence.

Chain-of-thought prompting asks a model to approach a problem through intermediate reasoning rather than jump directly to an answer. Task planning selects actions that can move a system from its current state toward a goal. The distinction is not whether the response contains numbered steps: it is whether those steps connect to available operations and their execution requirements.

For each workload request, IAM evaluates the principal, action, resource, and request context against applicable policies. A matching explicit Deny wins over any Allow. Without a sufficient Allow, the result is an implicit deny. Evaluation is not a first-match list: adding another Allow cannot override an explicit Deny.

A least-privilege AI integration starts with an access contract: which workload assumes the role, which model operations it performs, which data it reads or writes, and under what conditions. Separate runtime inference, ingestion, and deployment when they require different capabilities. A runtime role should not inherit model administration or data deletion merely because another stage needs them.

Model routing selects which model should handle a request or workflow step. Ensembling uses multiple models’ task contributions to determine a prediction, answer, or decision. The important boundary is not how many models are deployed—or even called—but what their outputs do.

A routing policy makes a constrained decision for a request or workflow step. First construct an eligible set: models that support the input modality, context size, required output interface, permitted processing location, and other mandatory conditions. A low price must never compensate for a failed hard constraint. If the set is empty, use an explicit no-route outcome rather than silently relaxing requirements.

A useful foundation-model ensemble combines complementary evidence, not just more text. Here, an extraction specialist identifies document facts while a policy specialist identifies applicable rules and exceptions. Both receive the same versioned source snapshot; their outputs jointly determine the result.

A human decision gate separates an agent's proposed action from permission to execute it. Route cases by explicit policy: high-impact actions can require review regardless of model confidence. Present the proposed action, source evidence, uncertainty, review reason, and model and policy versions. Approval applies to that specific proposal and evidence version, not to whatever the agent produces later.

Amazon SageMaker Ground Truth manages dataset annotation: a labeling job turns source records into labeled records for training or evaluation. Amazon Augmented AI (Amazon A2I) manages human review associated with inference: a human loop collects judgments that an application can use when handling a model result.

A custom A2I review loop is an application-invoked human task, not an automatic FM confidence filter. Define the evidence reviewers see, the allowed decisions, and the fields the agent will consume: for example, decision, reason, and correctedAnswer. Create a worker UI with CreateHumanTaskUi, using a Liquid template and named form controls inside crowd-form. The template reads values from task.input; control names shape the submitted answers.

An asynchronous callback is a later, independent request that supplies a result to work already in progress. The original API request starts a review workflow and returns promptly, for example with 202 Accepted and an application review ID. It does not wait for the human decision.

A model's tool call is a proposal, not permission to execute. Function definitions describe arguments to the model; they do not prove that an invocation satisfies the contract. The tool handler must independently validate every invocation, including calls arriving through another client. Strands tool definitions and lifecycle hooks can support validation, but must not be the only enforcement point.

An MCP server implements protocol operations such as initialization, tool discovery, and tool calls. Hosting it is not just exposing an arbitrary HTTP function: the implementation must preserve the protocol behavior its client expects. This comparison uses request-driven Lambda Event Functions; other Lambda compute modes require their own lifecycle evaluation.

A tool's function log cannot explain why the model chose it or whether its result helped. Start a task trace, record model turns and proposed tool calls, then instrument dispatch, execution, result delivery, and task completion. Record a proposal even when it is rejected before execution. Use explicit application instrumentation: infrastructure tracing alone does not know that a model proposed a tool or consumed its result.

A stateless MCP tool handler can handle a request without remembering an earlier request in its own process. Its inputs include tool arguments and explicit request context; durable task, conversation, and business records live outside the handler. Interchangeability means another authorized instance running compatible code can continue from those records, not that every retry must return identical live data.

An ECS service maintains a desired number of tasks from a task definition. A task is a replaceable execution unit, not a durable home for an MCP session. For a remote MCP server using HTTP, keep the public endpoint stable while task addresses change underneath it.

An inference baseline starts with offered demand: the work callers attempt to submit, including work rejected or throttled. Successful completions alone can hide a capacity shortage. Measure arrival rates over intervals short enough to expose bursts, alongside input and output token distributions, concurrent requests, retries, and latency. Separate original requests from retry attempts so overload-driven retries do not become a false business-growth forecast.

An LLM server can have spare arithmetic capacity and still be unable to produce more tokens. Separate memory capacity, which determines what fits, from memory bandwidth, which determines how quickly resident data can reach computation. GPU memory must accommodate weights, the attention key-value cache, temporary workspaces, and runtime allocations—not just the model checkpoint.

KV caching retains attention keys and values so decoding need not recompute them for earlier tokens. Prefill populates the prompt cache; decode reads retained state and appends new state. For full-context attention, longer sequences still increase attention-related memory traffic. KV here means attention tensors, not a general-purpose key-value database.

Batch inference is a workload execution pattern: a collected dataset is processed as a job, and predictions are saved for later consumption. The consumer accepts deferred results. A job can start when data is ready; a fixed timetable and a persistent prediction endpoint are not required.

Continuous batching makes scheduling decisions between generation iterations, rather than holding batch membership fixed until every request finishes. In ordinary autoregressive decoding, each scheduled sequence typically advances by one token per iteration. Its next token still depends on its previous tokens; batching does not remove that dependency.

Lazy loading separates a live inference process from a model that can generate tokens. Track each immutable model version through ABSENT → LOADING → READY → EVICTING → ABSENT, with a failure path out of LOADING. A running control interface is not evidence of model readiness. Lazy container image fetching and memory-mapped host files do not, by themselves, provide demand-paged GPU weights.

An LLM serving image contains the inference engine, framework, CUDA user-space libraries, and compiled attention or quantization kernels. The NVIDIA driver belongs to the host; the container runtime exposes GPU devices and required driver interfaces. Installing CUDA libraries inside an image does not provide a working host driver or make a GPU visible.

A model fits only when weights + KV cache + runtime workspace + safety margin fit on every participating GPU. Weight storage starts at parameter count × bytes per parameter, but quantization also needs scales, metadata, and compatible kernels. Quantizing weights does not automatically quantize the KV cache. Host RAM must separately accommodate the loader, tokenizer, and any temporary CPU copies.

An LLM pod has two distinct memory budgets: host RAM and GPU device memory. For a GPU-resident model without CPU offload, device memory must hold model weights, the attention KV cache, and runtime allocations such as activations, workspaces, and allocator overhead. Weight storage starts roughly at parameter count multiplied by bytes per parameter; quantization changes this estimate but also introduces metadata and runtime-specific overhead.

On Amazon EKS, Running is not the same as ready for inference. A container can have a live process while its model files are incomplete or its weights are not loaded. Separate the lifecycle into artifact staging, runtime initialization, and admission to serving traffic.

A small language model has comparatively modest parameter and compute requirements, not a universal parameter cutoff. Deployment fit starts with a workload contract: supported tasks, quality floors, safety requirements, and serving limits. A standalone model must satisfy that contract across the supported workload; a strong average can hide an unacceptable failure category.

A conditional model cascade tries a small model first, validates its candidate, and invokes a more capable model only when the candidate fails acceptance checks. Unlike direct routing, the decision uses an actual first-stage result. Unlike outage fallback, escalation can happen after a technically successful API call.

Right-sizing inference means choosing the least resource-intensive model that meets a task's quality and latency requirements—not choosing the smallest model unconditionally. Benchmark smaller pretrained models on representative requests, including difficult examples, languages, context lengths, and required output formats. Parameter count alone does not establish task quality or serving efficiency.

Loose coupling means the legacy application depends on a business contract, not a model's prompt format, location, or availability. Here, a case update commits without waiting for a summary. The application retrieves the result through a separate API later; accepting work does not mean inference has completed.

An AI retrieval store is a derived projection, not the owner of enterprise data. This propagation-only architecture follows committed document changes into a searchable representation. Updating that representation does not update a foundation model's weights.

An independently deployable GenAI extension owns a business capability such as createSummary, not a public wrapper around a particular model. Its service contract defines accepted inputs, authorization, result structure, error meanings, and completion behavior. Prompts, model identifiers, and provider response formats stay behind that boundary.

A webhook pushes an event to a callback URL. HTTPS protects the connection, but does not prove that the caller is the expected producer. For a producer using signed notifications, follow its documented algorithm and verify the signature over the original request bytes. Preserve those bytes through the gateway integration; parsing and reserializing JSON can change what was signed.

Start with a responsibility, then describe its actions, resources, and conditions. “AI developer” is too broad. “Invoke approved models in development” is a usable responsibility. Keep model invocation, data curation, and access administration separate; combine permissions only when the actual task requires them.

Federation lets AWS accept evidence from an enterprise identity provider without receiving the user's password. The provider authenticates the identity and issues a signed assertion or token. AWS must validate that evidence before accepting its claims; a readable token is not necessarily a trusted token.

AWS IAM manages persistent identity and authorization configuration: users, groups, roles, and policies. An IAM role is an assumable identity with a trust policy and permissions policies; it is not a set of credentials and has no long-term access keys of its own.

Temporary credentials separate a bounded authentication lifetime from the permissions of a role session. AWS STS returns an access key ID, secret access key, session token, and expiration. Signed requests need the session token as well as the key pair; all three credential values must be protected.

Federation trust decides which enterprise identities may obtain a role session; it does not authorize model invocation or data access. Before federation, register the enterprise identity provider in IAM and configure the role’s trust policy. A valid signature is necessary but insufficient: the policy must select the trusted provider, the federation action, and the intended audience or identity claims.

AWS IAM Identity Center centralizes workforce access: which employees and groups can enter which AWS accounts or assigned applications. Use it to connect workforce identity management to AWS access rather than maintaining separate IAM users for employees in each account. An organization instance supports centralized AWS account access; an account instance does not manage AWS account access.

For federated workforce access, use an IAM Identity Center organization instance. Connect the enterprise identity provider using SAML for authentication, and provision users, groups, and memberships using SCIM when supported. A successful SAML sign-in does not itself create an AWS account assignment. Enforce enterprise authentication requirements, including MFA, at the external identity provider.

IAM authorization determines whether a principal may perform an AWS API action on the relevant resources. For example, permission for athena:StartQueryExecution allows a role to submit a query. It does not establish which governed tables or columns that role may read.

Data residency asks where specified data is stored or processed relative to an approved geographic boundary. Data sovereignty asks which legal authorities and obligations apply to the data and its handling. Residency requirements can come from contracts or internal policy as well as law; sovereignty is not simply a stricter residency setting.

A hybrid application's location is only the start of a data residency assessment. Define the approved geography for each data class, then identify where prompts, retrieved context, outputs and their derived copies are processed or retained. A request processed outside the boundary is a transfer even if no long-term copy is kept there.

Define a data-flow contract with legal and compliance teams before choosing infrastructure. For each data class, record permitted processing locations, permitted recipients and administrators, retention rules, and approved recovery destinations. Include prompts, retrieved context, embeddings, responses, caches, logs, and backups. A retrieval operation can export restricted data even when the original database never moves.

AWS Outposts places AWS infrastructure in a customer-selected facility, such as an on-premises data center or colocation site. AWS Wavelength places AWS compute and storage in telecommunications partners’ data centers, within their networks. Both extend AWS beyond a Region, but they bring processing closer to different things: Outposts to local systems, Wavelength to users reached through the carrier network.

A cloud AI application needs two approved destinations: an on-premises records service and a regional foundation model endpoint. This IPv4 design uses non-overlapping networks. The diagram shows request direction only; the VPC routing card represents a routing decision, not another appliance. Label the shared first leg Request packets, every records-branch leg Records requests, and both model-branch legs Runtime HTTPS.

An isolated foundation-model application still needs explicit paths to its dependencies. For regional InvokeModel calls, select the Bedrock Runtime interface endpoint, not merely the Bedrock control-plane endpoint. Interface endpoints use AWS PrivateLink and private network interfaces in your subnets. The managed model remains outside your VPC.

An on-premises workload reaches Amazon Bedrock through a regional interface endpoint: private network interfaces in your VPC, backed by AWS PrivateLink. For runtime inference in this example, create com.amazonaws.eu-west-1.bedrock-runtime. Management APIs use a separate bedrock endpoint; one endpoint does not cover every Bedrock API family.

This design uses an Outposts rack: a local records system sends data to an EC2 application for filtering before an approved prompt reaches a Regional foundation model. The arrows show forward payload flow, not every packet, response, or management connection.

AWS Wavelength places supported compute and storage in a telecommunications partner’s network. A subnet in a Wavelength Zone belongs to a regional VPC, but its physical location differs from the Region’s Availability Zones. Place latency-sensitive inference there; a regional foundation model call still incurs a regional round trip.

A GenAI gateway exposes an application contract, such as an approved summarization task, rather than unrestricted provider parameters. Here, Amazon API Gateway uses IAM authorization for signed client requests. Elastic Load Balancing forwards application traffic, and AWS WAF filters web requests; neither supplies API Gateway's IAM-authorized API routes.

API Gateway uses a token bucket for request throttling: the rate setting replenishes request-admission tokens, while the burst setting bounds bucket capacity. A short spike can consume accumulated capacity faster than it replenishes. When admission capacity is unavailable, API Gateway can reject the request with 429 Too Many Requests, rather than queue it for later. Burst capacity is not a limit on in-flight backend executions.

A security quality gate turns security evidence into an enforced release decision. This diagram shows a container release's evidence and decision flows—not invocation calls, lookup requests, or image downloads. Its policy verifier and exception handling are custom implementation choices, not automatic features of a scanning service.

A quality gate must translate unacceptable results into a failed build, not merely print warnings. Keep the buildspec, test harness, and policy thresholds under version control together. With version: 0.2, commands share shell state across commands, unlike the isolated command shells of version 0.1.

An SDK retry policy decides whether a failed model API call is eligible for another attempt and how to delay that attempt. Standard mode combines retryable-error classification, exponential backoff with jitter, and a retry quota that limits retries during sustained failures. It is a good starting point when you want retries without adaptive client-side request pacing. Authentication failures and invalid requests generally require correction, not repetition.

A queued model API returns acceptance, not an inference result. Validate the request, create a stable tenant-scoped job ID, and confirm that SQS accepted the message before returning an accepted response. Include the job ID, input or input reference, model selection, parameters, and a request fingerprint. If the send outcome is uncertain, retry with the same job ID; duplicate messages are safer than falsely reporting acceptance.

Amazon SQS stores invocation requests until workers can process them. Accepting a message is not completing an invocation: the worker must receive it, call the model, persist the result, and explicitly delete the message. This design uses a Standard queue, so workers must tolerate duplicate deliveries and out-of-order requests.

A request contract defines accepted inputs, not just a JSON shape. For an inference endpoint, separate required request parameters, body structure, and rules that depend on the selected model or caller. API Gateway REST APIs can reject basic violations before invoking the integration; do not assume HTTP APIs offer the same built-in validation.

Server-Sent Events (SSE) carries server-to-client text events in a long-lived HTTP response. The client initiates the request, but that response is a one-way application channel. Another prompt, feedback, or cancellation requires a separate request or another mechanism. This is often sufficient for an assistant that receives a prompt and then displays incremental text.

Chunked transfer encoding lets an HTTP/1.1 response carry generated output before the total body length is known. A streaming-enabled API integration supplies body bytes incrementally; the HTTP layer frames those bytes for transmission. The response declares Transfer-Encoding: chunked and must not also send Content-Length. HTTP libraries normally handle this framing rather than application code constructing it manually.

A streaming backend is a protocol adapter. Amazon Bedrock ConverseStream supplies structured SDK events, not browser-compatible SSE. The backend selects useful output and encodes a separate HTTP response. ConverseStream is a Bedrock API; a SageMaker inference endpoint uses a different API contract.

Server-Sent Events (SSE) defines application events in a UTF-8 response with the media type text/event-stream. A data: field contributes event content; a blank line ends the event block and triggers dispatch when the block contains data. Neither the media type alone nor a transport setting creates those fields and delimiters.

The browser's WebSocket connection and the model's response event stream are separate interfaces. A backend reads model events, converts them into application messages, and sends them to the connection. API Gateway does not automatically translate Bedrock output into browser messages.

Response streaming sends partial output within one HTTP response. API Gateway REST APIs support streaming responses through supported Lambda proxy and HTTP proxy integrations. Configure the integration's responseTransferMode as STREAM and use a streaming-capable backend; a normal buffered Lambda return does not become a stream merely because it contains several strings. Do not assume that HTTP APIs have the same streaming support as REST APIs.

A model API failure must pass two gates before another attempt: is the failure eligible for retry? and is replay acceptable? Use the API's documented error codes and the SDK's retry classification. Throttling, selected transient server errors, and some transport failures are candidates; malformed prompts, unsupported parameters, and authorization failures generally require correction instead.

Inference admission must often satisfy both a request-rate budget and a token-rate budget. Each invocation consumes one request allowance, but its token cost depends on the work counted by the provider. Many short invocations can exhaust request allowance; a few long invocations can exhaust token allowance. Spare capacity in one budget cannot compensate for exhaustion of the other.

Exponential backoff answers: after an eligible failure, when should this operation try again? Its delay ceiling grows with successive failures, usually up to a cap. Jitter randomizes the wait so clients do not repeatedly retry together. These mechanisms react to an attempt that has already happened.

A slow or incorrect foundation model (FM) response creates different troubleshooting questions: Where did this request spend its time? How widespread is the change? What did the application actually send and validate? These require different evidence, even when one observability console brings the signals together.

OpenTelemetry instrumentation creates spans around application work and outbound calls. In this architecture, a frontend calls an inference service, which invokes an FM through Amazon Bedrock. Instrumentation observes the call; it does not require recording its prompt or response body.

A content-based router selects a foundation model from request attributes, not merely endpoint load. This implementation separates request normalization, attempt orchestration, and application-level model selection. The diagram shows outbound calls and telemetry; responses and internal workflow transitions are omitted.

A context window is a model constraint: how much information a request can contain under that model's accounting rules. An application budget is a policy constraint: how many tokens or how much money the application permits. Passing the model's capacity check does not imply passing either application check.

A failure message identifies a symptom, not necessarily its source. Diagnose an individual attempt by pairing its outgoing request evidence with its response or transport exception. A logical operation may contain several SDK retries: the final exception alone can hide earlier throttling or a different failure. Use an application correlation ID across the operation, a separate attempt identifier, and the provider request ID when one is returned.

A consumer-driven contract starts with what a client must accomplish and the decisions its interface must support. For a support-case summarizer, agree on POST /summaries with caseId, language, and lengthPreference, rather than copying a provider's invocation payload. Review examples with consumers before implementing the backend. Keep model identifiers, prompt templates, and inference settings private unless consumers have a genuine requirement to control them. The backend translates application intent into model-specific calls.

A declarative component describes the interface for its current inputs. Compose a conversation from reusable parts: a transcript, composer, model-readiness indicator, response controls, error panel, and announcement region. AWS Amplify UI can supply building blocks, but the application still owns their state contract and accessibility behavior. Keep conversation data above these components; pass state down and send user actions back through callbacks.

Amplify UI AIConversation is a declarative React interface, not a model endpoint. In the Amplify AI Kit conversation pattern, the browser binds to an authenticated Amplify Data conversation route; a backend handler invokes Amazon Bedrock. The diagram shows this interaction path, not every resource used for conversation persistence.

A durable orchestrator owns the business process, not the business operations themselves. AWS Step Functions records execution progress and state between tasks, evaluates conditions, and decides what happens next. A Lambda function can update a CRM record; it should not have to remain running while the process waits for extraction or approval.

The supplied Amazon Q lifecycle notice states that maintenance began on June 30, 2026: new customers are not accepted, and existing customers can continue using it. AWS’s migration guide is https://docs.aws.amazon.com/amazonq/latest/qbusiness-ug/qbusiness-availability-change.html. That guide concerns Q Business; it does not establish Q Developer’s lifecycle status. This lesson describes capabilities AWS offered, rather than recommending a new deployment.

An AI coding assistant can produce valid SDK code that is wrong for your application. Grounding means supplying authoritative project constraints, not merely asking for similar-looking code. For an assistant such as Amazon Q Developer, explicitly select or attach the relevant files using the context controls available in your environment. Do not assume that an open workspace means every relevant file was retrieved or read.

Amazon Q is in maintenance since 2026-06-30: it is closed to new customers, while existing customers can keep using it. AWS's migration guide is https://docs.aws.amazon.com/amazonq/latest/qbusiness-ug/qbusiness-availability-change.html. This lesson teaches Q Developer as AWS offered it, not as a recommendation for new workloads.

Amazon Q Developer, which incorporates the code-suggestion capabilities formerly called CodeWhisperer, uses development context to propose code. Context guides generation; it does not prove correctness. Nearby comments, function signatures, imports, and existing implementations help steer inline suggestions. When requesting a larger refinement in chat, explicitly supply the relevant files or code rather than assuming every repository detail is available.

A deterministic GenAI unit test asks: given this model response, does the application behave correctly? It does not ask whether the model will produce that response. Even a low-temperature live model call is not a substitute for a fixed test input.

Refactoring changes code structure while preserving observable behavior. Extracting model invocation into an adapter can be refactoring; changing the model, improving a prompt, adding caching, or introducing retries is a separate behavior change. Keep these changes separate so a failing comparison has a small set of possible causes.

An AI-assisted refactor should change implementation without changing approved behavior. For a GenAI component, that behavior includes more than the final answer: prompt roles, retrieved context, tool arguments, response schemas, refusal handling, retries, and error mapping can all be observable contracts. Ask Amazon Q Developer for a bounded implementation change, not an unrestricted improvement, and define what must remain invariant before accepting its patch.

A request timing measures elapsed time; a CPU profile shows where sampled execution used CPU. A slow request can therefore have little CPU activity. Instrument retrieval, prompt construction, model invocation and response processing as spans, and propagate trace context across supported boundaries. Record retries, connection-pool acquisition and backoff separately where possible.

An FM application can retry an invocation, call several tools, or run model calls in parallel. Reconstructing its failure sequence requires an operation key that survives those actions, not just nearby timestamps. A useful application-defined key is (environment, tenant_id, operation_id). Propagate it through application logs and queued work; a conversation ID alone can merge separate user turns.

An FM request can include retrieval, independent classification, prompt assembly and generation. Its end-to-end latency is the elapsed time to a defined completion boundary, not the sum of every operation's duration. Here, completion means that the orchestrator has finished returning the generated response.

Event correlation relates observations using evidence such as shared identifiers, resources, attributes, and timing. The relationship is inferred by a query, rule, or investigator; it need not have been recorded when the activity happened. Collecting events together is aggregation, not correlation.

Deterministic checks enforce exact request contracts: required fields, permitted properties, types, lengths, nesting limits, and allowed operation values. Decode and normalize consistently before applying relevant rules. Reject ambiguous representations rather than letting different validators interpret different inputs.

Text-to-SQL moves arithmetic from model-generated prose into a database. The model proposes a query; the database computes its result. But “net revenue by region last month” is not yet a calculation: it leaves the revenue definition, reporting period, region attribution, and aggregation level unresolved.

A constrained text-to-SQL application maps language to a supported intent and candidate values. For example, the model proposes sales_since and a start date. Trusted application code selects a reviewed query structure; it never forwards model-written SQL or falls back to unrestricted SQL when no intent matches.

Nondeterminism concerns whether repeated executions produce different outputs under the same effective inputs and configuration. Hallucination concerns whether an output contains false, fabricated, or unsupported information. Variation is a relationship between runs; factual support is a relationship between a claim and the evidence it should use.

Claim-level verification asks whether each factual assertion follows from evidence the application is allowed to trust. Split a draft into independently checkable claims, preserving qualifiers such as product version, date, region, and eligibility conditions. One sentence can contain several claims: “Standard-plan customers receive refunds automatically” asserts both eligibility and an automatic process.

An indexed passage is a derived artifact, not a copy you can assume is faithful. Evidence lineage links an immutable source version to extracted text, chunk boundaries, the exact embedding input, and the stored index record. A successful ingestion job proves that work completed; it does not prove that the passage survived correctly.

A retrieval test needs labeled reference passages, not just expected answers. For each query, record the evidence that would support an answer, its source identifier and version, and any required region, date, or permission scope. Label passages for their ability to support the requested fact: a high similarity score or shared vocabulary is not a relevance judgment.

A useful evaluation case contains more than a question and an ideal sentence. Record the user's task and scope, authoritative reference passages with versions, required facts, acceptable alternatives, and expected behavior when evidence is missing. Build representative cases from real task categories, including multi-part questions, conflicting or outdated sources, unanswerable requests, and permission-sensitive requests. Include rare, high-impact cases deliberately; report them separately rather than allowing frequent easy questions to hide their failures. Keep a held-out set separate from cases used to tune prompts or evaluation rules.

A claim-support score measures whether specified evidence supports a claim—not whether the claim is universally true. Split a generated answer into independently checkable claims, preserving conditions, dates, quantities, and exceptions. For each claim, retain the supporting passage and document version, then distinguish supported, contradicted, and insufficient evidence. Missing evidence is not evidence that a claim is false.

A golden dataset is a reviewed set of inputs, authoritative evidence, and expected behavior. For diagnosis, each case needs more than an ideal answer: record required claims, their supporting passages or tool results, relevant dates and scope, and when the system should abstain. This separates an unsupported assertion from a valid paraphrase or a justified refusal.

DetectPiiEntities returns candidate PII spans with a Type, Score, BeginOffset, and EndOffset. Types such as NAME, EMAIL, and PHONE describe what was detected; they do not decide what your application must remove. Comprehend PII detection supports English and Spanish, not every language supported by other Comprehend features.

Defense in depth assumes that every safeguard can miss a violation. Input controls reduce what reaches the model, model safeguards apply configured content policies, and output controls decide what the application may release. A system prompt guides generation; it is not an authorization boundary.

Jailbreaking targets a model’s safeguards: the attacker tries to defeat protective restrictions on what the model will disclose, generate, or support. Prompt injection uses attacker-supplied instructions to redirect application behavior away from its authorized task or governing instructions. Jailbreaking emphasizes the restriction being attacked; prompt injection emphasizes the instruction-based mechanism.

Sanitization changes untrusted input for a specific processing context. Validation decides whether input meets requirements. Rejecting a field containing prohibited control characters is validation; removing permitted-to-remove control characters is sanitization. Neither operation proves that the resulting text is safe to follow as an instruction.

Prompt-injection detection asks whether content is trying to gain authority it does not have. Screen three entry points: user messages, retrieved passages, and tool responses. A user can request an allowed task, but cannot override application rules. Retrieved passages and tool responses supply evidence or results, not new application instructions—even when they arrive through an authenticated service.

A jailbreak detector estimates whether input is trying to defeat model safeguards. For a single-turn request, useful signals include claimed authority to override restrictions, requests to suspend safeguards, and attempts to disguise a prohibited objective. These are evidence, not proof: a legitimate security discussion can describe the same techniques. Judge the requested behavior and its context, not just the presence of suspicious words.

Prompt injection attempts to turn untrusted text into instructions that override application policy; a jailbreak attempts to bypass model safeguards. Detect the attempted instruction change, not merely words such as “ignore” or “system.” A legitimate security lesson can quote those words without requesting a bypass.

Red teaming searches for violations of a defined security or safety requirement, not merely poor answers. Obtain written authorization identifying target environments, identities, permitted attack surfaces, request and spending limits, stop conditions, and an owner who can halt execution. Use synthetic secrets, isolated tenants, restricted credentials, and simulated transactions.

An automated adversarial evaluation needs a test contract, not just difficult prompts. Define the protected asset, forbidden outcome, attacker-controlled surface, attacker knowledge, and interaction budget. Distinguish direct user instructions from attacker-controlled retrieved documents or tool responses. Inject each test through its actual surface: putting a malicious document into the user message tests a different trust boundary.

An AI application can call a managed foundation model without an internet path. This design permits two application API paths: regional Amazon Bedrock Runtime inference and application log writes to CloudWatch Logs. The arrows show outbound API requests; responses, DNS exchanges, and configuration controls are not separate arrows. The model remains a managed service outside your VPC.

A Lake Formation SELECT grant defines what a principal may read from a cataloged table. A table-wide grant permits all its data; a column-level grant limits the permitted columns. Access to a database or table's metadata is not itself permission to read its records.

Data Loss Prevention (DLP) makes a transfer decision about content, identity, purpose, and destination. Define the protected data and permitted boundaries first: permission to read a customer record does not automatically permit sending that record to a model, another tenant, or an external tool.

Amazon Macie discovers sensitive data in S3 objects; it does not automatically inspect every dataset a foundation model can use. Start by mapping training, fine-tuning, evaluation, and retrieval-source data to their S3 buckets and object prefixes. Include intermediate exports and copies where they belong to the privacy review. Macie configuration and discovery are Regional, so account and Region boundaries matter.

Amazon Comprehend inspects text that an application supplies. Its PII detection can support a decision about a particular message before that message reaches a foundation model. Amazon Macie discovers sensitive data in S3 and reports findings tied to stored data. It supports decisions about a data estate, not synchronous admission of a prompt.

For sensitive AI prompts, responses, and interaction logs, define the retention clock before configuring cleanup: which event starts it, which copies it covers, and which exceptions allow longer retention. S3 Lifecycle age-based expiration uses object creation time, not an interaction timestamp inside the object. Exporting an old conversation into a new object can therefore start a different clock.

A purpose contract specifies the task, allowed fields, necessary precision, permitted recipients, retention, and a minimum utility target. For a support-summary task, allow product, error code, and relevant troubleshooting text; exclude payment details and unrelated account history. Reject unexpected fields before downstream calls rather than accepting everything and cleaning up later.

Detection-driven masking locates sensitive spans and replaces selected values before text crosses a defined boundary. Specify the forbidden values, the permitted context, and the processing services allowed to receive raw text. A detector must see the original text; masking before a foundation model does not mean the detector never receives it.

Anonymization must be evaluated against an attacker’s plausible knowledge in a specific release context. For AI support-ticket data, an attacker might know a person’s occupation, town, and incident date from a public post. Those attributes can link the person to a ticket even after names and email addresses are removed. Treat combinations of fields, narrative details, and event sequences as possible quasi-identifiers, not just individual columns.

Data masking obscures selected values. Data anonymization aims to prevent information from being associated with an identifiable person in its intended release context. These describe different things: masking is a transformation; anonymity is a property of the resulting information and the ways people can use it.

Data minimization asks whether information is necessary for a specific purpose, at the required precision, for particular recipients and for a justified duration. Data masking changes how selected values are represented or disclosed. These controls answer different questions: whether information belongs in a workflow, and how sensitive values should appear where information is used.

Data minimization asks whether information is necessary for a defined purpose. The decision includes which records and fields are needed, their precision, who receives them, and how long they persist. Necessary information is not necessarily anonymous: completing a delivery can require an identifiable address.

Data provenance describes a source's origin and relevant processing history. It connects an artifact to evidence about collection, ownership, permissions, and transformations. Source citation identifies material offered as support for a generated claim. Provenance asks how the evidence came to exist; citation asks which evidence supports this statement.

Source identity must survive changes in representation: an object becomes cleaned text, chunks, retrieved context, and input to generation. Pair a stable logical source ID with an exact version locator, such as an S3 bucket, key, and VersionId. A path alone can resolve to different bytes after an update.

The AWS Glue Data Catalog stores descriptions of datasets, not copies of their contents. For source attribution, record the table’s Region, CatalogId, DatabaseName, and Name. A table name alone is ambiguous across databases, catalogs, and Regions. For the default Data Catalog, the catalog ID is the AWS account ID. Keep this qualified identity separate from the underlying storage location.

An evidence-backed model card connects claims about a model to inspectable records. Document intended tasks, users and operating conditions; unsuitable uses; evaluation methods and results; known limitations; and accountable owners. For a foundation-model application, distinguish the provider's model documentation from your evidence about the deployed application, including its prompts, retrieval configuration and safeguards.

Amazon SageMaker AI model cards hold structured documentation for a model: its identity, intended use, evaluation results, and known limitations. For a foundation model, identify the exact model revision and any customization being reviewed. A card records the basis for a compliance decision; it does not certify compliance or enforce runtime behavior.

A durable citation needs a logical source ID, an exact revision, and a passage locator. For example, policy-17 identifies a document across revisions; an S3 bucket, key, and VersionId identify one stored revision; a page and section identify supporting material within it. A key alone resolves to the current object, not necessarily the bytes used for an earlier answer.

A foundation-model governance framework governs a use case, not just a model. Record the intended purpose, affected people, data, model provider, deployment context, and permitted actions. The same model can require different oversight when drafting internal notes and when influencing consequential decisions. Assess inherent risk before use-case controls, considering potential harm, exposure, autonomy, and reversibility.

Independent approval means that the actor proposing a foundation model change cannot authorize that same change. The controlled change includes more than model weights: model selection, system prompts, guardrail settings, retrieval sources, and deployment configuration can all change production behavior.

Policy as Code turns organizational AI requirements into executable checks. A versioned bundle contains rules, input schemas and supporting policy data. Checks can require an approved model identifier, permitted Region, configured guardrail and a reference to an approved evaluation. They verify declared facts and trusted evidence, not the safety of every future model response.

Service control policies (SCPs) set the maximum permissions available to covered principals in member accounts; they do not grant permissions. SCPs require AWS Organizations all features and the SCP policy type to be enabled. Policies attached to the organization root, each parent organizational unit (OU), and the account all participate in evaluation.

AWS Service Catalog separates centrally approved infrastructure from local self-service launches. This example shares a portfolio containing a CloudFormation product for a private, encrypted bucket used by foundation model workloads. Sharing distributes product definitions and constraints; it does not deploy resources into recipient accounts.

Anomaly detection asks whether an observation departs from expected behavior for its context. The observation can be one request, a time-window metric, or a batch statistic. A token-usage burst can be anomalous relative to an expected range without saying anything about the distribution of prompt content.

A contextual baseline estimates expected behavior for a particular workload and time. Define stable workload cohorts, such as task type, model version, and prompt version. Otherwise, a shift from short classification requests to long document summaries can look like abnormal token use even when both tasks behave normally. In CloudWatch, express this context through appropriately separated metric series; the detector does not automatically infer task or prompt context.

A drift score is meaningful only relative to a defined reference. A versioned baseline binds an accepted dataset to its sampling period, cohort definitions, transformations, metrics, and threshold policy. Record model, prompt, retrieval configuration, embedding model, and evaluator versions where they affect measurement. A baseline is therefore more than a stored average: it specifies how to reproduce a comparison.

A group-level baseline records how a defined system behaved on a defined population. A fairness requirement states what behavior is acceptable. They are not interchangeable: a historically unequal system can remain stable while still violating the requirement. Monitor both absolute disparities and changes from the baseline, rather than treating “no drift” as “fair.”

Continuous policy assurance asks whether a deployed application still meets its policy, not merely whether its filter is enabled. Create a versioned evaluation corpus containing prohibited requests, legitimate near-boundary requests, sensitive-data examples, and multi-turn attacks. Attach expected outcomes and scoring rules to each case. Include allowed cases so that blocking everything cannot count as success.

A useful user-facing explanation identifies the AI-generated answer, the evidence supporting its material claims, and its limits. A generated rationale is another model output: fluent wording does not establish that it faithfully describes the model’s internal computation. Prefer “This recommendation is supported by these passages” over “These are the exact steps the model used to think.”

Fairness is a context-dependent requirement, not a property established by one model score. Define the task, affected populations, and harm with domain specialists and affected stakeholders. For a help assistant, relevant harms include unjustified refusals, lower-quality assistance, and demeaning stereotypes. Measure these separately: equal refusal rates could conceal equally poor service, and toxicity scores do not measure unequal assistance.

LLM-as-a-Judge applies an evaluation prompt to a candidate answer. The prompt must define the task, criteria, permitted evidence, and output format. Its score describes compliance with that contract—not objective quality in every context. For a retrieval-augmented answer, provide the original question, candidate answer, and exact retrieved passages with stable identifiers. A reference answer is optional; evidence is not optional when judging support against those passages.

A rubric defines what a score should mean; it does not determine how an evaluator reaches that score. A human reviewer interprets criteria through knowledge, experience, and incentives. An LLM judge applies them through learned model behavior and the supplied context. Identical instructions therefore do not make the two judgments interchangeable.

A foundation model configuration includes the model version, prompt template, decoding parameters, retrieval settings, tool policies, and safety controls. To estimate the effect of one change, freeze the other settings. For example, change only the prompt while holding temperature, top-p, retrieval corpus, and token limits constant. Changing several settings together is valid, but the result estimates the combined intervention, not each setting's contribution.

A/B testing asks whether a defined alternative improves an outcome relative to a control. Canary testing asks whether exposing more production traffic to a candidate is acceptable under release criteria. Both can split live traffic between the same configurations. Neither the split percentage nor the routing service determines which question the evidence answers.

A model card is a structured, versioned record of a model’s intended uses, evaluation results, limitations, and ethical considerations. For a suitability decision, identify the exact model version, accountable owner, intended users, languages, modalities, and use boundaries. Record unavailable information, such as undisclosed training-data details, as unknown rather than filling gaps with assumptions.

Model-aware token accounting measures the complete model input, not just the latest user message. Include system instructions, retained conversation turns, retrieved evidence, tool definitions, tool results, and the message structure that the provider converts into model input. A count of visible text alone can miss model-specific formatting overhead.

Text chunking groups text into passages used as processing or retrieval units. Tokenization converts text into a sequence of token identifiers using a model-specific vocabulary and encoding rules. A chunk is an application-level unit; a token is a representation unit, not necessarily a word, character, or meaningful passage.

A workflow allowance covers every model call needed to complete a task: planning, answering, tool-call follow-ups, retries, and model-based summarization. Track input and output separately. Reusing history in another request consumes input tokens again unless the provider applies a different accounting treatment, such as supported prompt caching.

Context pruning selects material to remove from a model request while preserving what the task needs. The context window is a hard constraint, not a target to fill. Establish the selected model and API’s input, output, and combined-window rules before choosing what to remove.

Prompt compression replaces a request's wording or representation with fewer tokens while preserving what the model needs to do the task. Treat it as a constrained optimization: minimize total cost subject to explicit quality requirements, not merely minimize prompt length. The prompt includes instructions, examples, history, evidence, and tool results.

Context pruning selects which material enters a model request. Prompt compression represents retained information more compactly. Removing an unrelated incident report is pruning; rewriting a retained report as a compact record is compression. Both reduce input, but they make different decisions: what to include versus how to express it.

A stop sequence makes the serving system terminate generation when generated text matches a configured string. It is a boundary detector, not an instruction to become concise. Prompting the model to append END_RESPONSE encourages that boundary; configuring the same string as an inference parameter enforces stopping if the model produces it. The model can still omit the marker or produce it too early.

Completed-job throughput means finishing useful datasets, not merely starting more requests. Compare runs with the same input snapshot, model, and generation settings, and count a job as complete only after its required outputs are durably written and checked. Equal-sized record partitions can have very unequal runtimes when prompt and generated lengths vary.

A configuration includes more than the model: prompts, retrieval, output limits, routing, retries, and human review can all change its value. Compare each candidate with a credible baseline, such as the current workflow or a rules-based alternative. Fix the business objective, evaluation period, and workload segments before measuring differences.

Cost per accepted result measures the cost of producing a usable task outcome, not merely generating tokens. Define acceptance before testing: for example, a result must be correct, grounded in supplied evidence, valid against a schema, and compliant with required instructions. Count each original task at most once, even if several attempts were needed. Lower cost per accepted result is better.

Price-performance ratio asks how efficiently a configuration delivers a defined capability. With accepted outputs per dollar, higher is better; with cost per accepted output, lower is better. The comparison must hold workload and acceptance requirements consistent. A raw benchmark score per dollar is weaker evidence when score improvements do not represent useful task improvements.

Foundation model requests are not equal units of work. Record each request’s prompt tokens, completion tokens, arrival time, model version, and generation settings. Preserve the joint distribution: long prompts and long completions occurring together can create pressure that separate averages hide. Include system instructions, retrieved context, and conversation history in prompt counts, using the model’s tokenizer.

Token-demand forecasting starts with inference calls, not just application requests. Include repeated calls, retrieved context, and conversation history. Segment demand by model and request class, preserving prompt/completion length pairs, bursts, and growth. Ten inference requests per second averaging 2,000 input tokens and 500 output tokens imply 20,000 input tokens per second and 5,000 output tokens per second.

For model inference, requests per second is an incomplete scaling signal. A short prompt with a short answer and a long prompt with a long answer impose different loads. Separate prefill, which processes prompt tokens, from decode, which generates completion tokens incrementally. Their throughput and resource demands differ, so one combined token count can hide the limiting phase.

Capacity planning determines what resources a future workload will require to meet its service-level objectives. Its outputs include a capacity baseline, growth and failure assumptions, headroom, and constraints that must be resolved before demand arrives. Planning is repeated as evidence changes; it is not just an initial sizing exercise.

Right-sizing determines which resource configuration can serve an inference workload efficiently while meeting its requirements. The configuration includes accelerator compute and memory, host resources, and serving settings. The smallest instance is not necessarily the best fit: a larger configuration can deliver more useful throughput per unit of cost.

Target tracking adjusts capacity to keep a utilization metric near a target; it is not simply a rule that adds one instance whenever demand crosses a threshold. For inference, the metric should express demand per unit of capacity. With steady demand and equivalent replicas, doubling usable capacity should approximately halve the metric.

An exact-response cache returns a previously generated answer without invoking the model again. Exact means equality under a deliberately defined request representation, not similarity of meaning. A deterministic key does not make model generation deterministic: caching freezes one accepted result. If the product promises a fresh independent sample on every call, bypass response caching.

A deterministic cache key represents an equivalence contract: two requests may share a stored result only when the application considers their effective inputs interchangeable. This is exact-request caching, not semantic similarity matching. Hashing only the visible user prompt misses dependencies; even a perfect hash cannot repair an incomplete identity.

A generated response is reusable only while its validity contract holds: the request is compatible, its dependencies are current enough, and the caller is still authorized. An exact request hash or a high semantic similarity score finds a candidate; neither proves that it is safe to return. This lesson concerns answer caching, not model prompt caching, which reuses supported prompt-prefix processing rather than a completed answer.

Inference response caching reuses a completed model response instead of invoking the model again. This design uses cache-aside: application code decides whether reuse is allowed, reads ElastiCache, and invokes the model only when necessary. ElastiCache does not decide whether two prompts mean the same thing.

Edge reuse avoids an origin model invocation by serving an already generated HTTP response. The essential contract is that requests sharing a cache key may safely receive the same response while it remains fresh. CloudFront does not inspect prompts for semantic equivalence or decide whether an answer is personalized.

A semantic response cache retrieves a stored answer using similarity between request representations, usually embeddings. Differently worded requests can therefore reuse one answer. But similarity measures closeness in a representation space, not answer interchangeability: “Can I cancel?” and “Can I cancel without a fee?” may be close while requiring different answers. A retrieved entry is a candidate, not yet a cache hit.

Semantic response caching retrieves a stored answer using similarity between request embeddings. Unlike an exact key, an embedding can connect differently worded requests. A hit avoids the generative model invocation, but embedding inference, vector search, and acceptance checks still run.

Prompt caching reuses model-side processing of a previously submitted input prefix. During prefill, a model processes input tokens and builds internal state used to generate an answer. A supported caching implementation retains reusable prefix state so a later invocation can avoid repeating some of that work.

The application critical path is the longest chain of required work between request arrival and a defined completion boundary. Here, completion means the full answer is ready. Model execution is only part of that interval: input preparation, retrieval, queueing, and communication can all delay it. Represent each operation by its duration and the results it must receive before starting.

A reusable inference artifact needs a reuse contract: which requests it answers, which source revision it represents, and who may read it. For a prepared product answer, identity can include product revision, language, tenant scope, prompt revision, model identifier and configuration, and output-schema revision. Record generation settings and provenance; recording them does not make model output deterministic.

Precomputation deliberately executes work before a request needs its result. The trigger might be a source-data change, a schedule, or a prediction of future demand. Generating product summaries when catalog records change is precomputation relative to later requests for those summaries.

Parallel execution overlaps branches whose inputs are already available and which do not need one another's results. Asynchronous remote calls can provide this overlap without multiple local CPU cores. A later synthesis that consumes their outputs remains a dependency and must wait.

Response streaming changes when available output is delivered. Inference acceleration changes when the model produces that output. Holding the model, prompt, generated answer, and serving conditions constant, exposing successive portions of an answer does not remove the computation needed to generate it.

A streaming model API does not guarantee visible streaming. Every stage between generation and the screen must release useful content before the answer finishes. This diagram shows only the response path; request routing and authentication are omitted.

A production stream needs an incremental producer, relay, transport, and reader. This architecture uses Amazon Bedrock ConverseStream with a model that supports it, an ECS application, and an Application Load Balancer that directly routes to registered task IP targets. The diagram shows response delivery and telemetry only; request submission is omitted.

For foundation model context retrieval, define retrieval latency from the moment the retrieval layer accepts a query until the selected context is ready for prompt assembly. This boundary includes preprocessing, query embedding, search, document fetching, reranking, and context selection when those operations are required. It excludes subsequent model generation. A search engine’s execution time measures only a nested portion of this interval.

At each generation step, the model produces logits, or scores, for possible next tokens. For positive temperature T, standard temperature sampling uses p_i(T) = exp(z_i/T) / sum_j exp(z_j/T). Equivalently, starting from probabilities at temperature one, p_i(T) = p_i(1)^(1/T) / sum_j p_j(1)^(1/T). These equations describe temperature alone, before any additional candidate filtering.

Top-K sampling restricts the next-token distribution, not the number of complete answers. At each generation step, the decoder retains the K highest-probability tokens, removes the others, renormalizes the retained probabilities, and samples one token. The generated token extends the context; the model then computes a new distribution and repeats the procedure.

Nucleus sampling acts on the next-token probability distribution at each generation step. Sort candidates from highest to lowest probability, then retain the smallest leading set whose cumulative mass reaches or exceeds top-p. The token that crosses the threshold stays in the set: the retained mass can exceed the requested threshold.

At each generation step, a language model produces logits, or scores, for possible next tokens. For positive temperature T, temperature sampling uses p_T(i) = exp(z_i/T) / sum_j exp(z_j/T). Lowering T increases the probability advantage of higher-scoring tokens; raising T reduces that advantage. With finite logits and positive T, temperature preserves token ranking and does not explicitly remove candidates.

Both controls act on next-token selection, but they change different things. Temperature reshapes probabilities. For logits z_i and positive temperature T, the distribution is q_i(T) = exp(z_i/T) / sum_j exp(z_j/T). Equivalently, if r_i is the probability at temperature 1, then q_i(T) = r_i^(1/T) / sum_j r_j^(1/T).

At each generation step, compare both methods on the same next-token distribution, holding other decoding settings fixed. Sort token probabilities as p1 ≥ p2 ≥ ..., using a consistent tie rule. Top-K sampling retains the first K candidates, assuming at least K candidates have positive probability. Their identities can change at every step; only the count is fixed.

Connection pooling separates a request's lifetime from its connection's lifetime. Reuse an HTTP client or retrieval-database pool across requests within the lifetime and concurrency rules supported by the library. Constructing and closing a client for every request can repeatedly incur DNS resolution, transport establishment, TLS, and database authentication work; the exact setup path depends on caching and protocol.

A queue converts a traffic burst into waiting work, not extra inference capacity. This architecture covers admission through durable completion or dead-letter handling. Its application handler returns a job identifier and 202 Accepted only after enqueueing succeeds; that response promises acceptance, not a successful generation.

Task completion rate is verified successful tasks / eligible attempted tasks × 100. The unit is the user's task, not a message, model invocation, or tool call. A task can cross several systems and still contribute only one outcome.

A decision-ready KPI connects a business outcome to a decision rule. Define its numerator, denominator, unit, eligible population, observation window, data source, owner, and direction of improvement. For a support assistant, use cases resolved without reopening within seven days divided by eligible cases—not model responses generated. Pair the outcome with quality, safety, and cost guardrails so that apparent efficiency cannot hide rework or harm.

AWS Cost Anomaly Detection uses machine learning to identify unusual spending from historical AWS cost patterns, including recurring patterns such as day-of-week variation. Its baseline is learned, not a spending ceiling you configure. A GenAI workload can therefore trigger an anomaly while still below its budget, or spend heavily without being anomalous if that spending matches its established pattern.

A searchable window defines how long investigators can query records immediately; total retention defines how long evidence must remain retrievable. The diagram uses illustrative requirements: 30 days in CloudWatch Logs and archive objects expiring after 365 days. These are design examples, not compliance defaults.

Integrity asks whether collected bytes still match a trusted baseline. Provenance asks where those bytes came from and how they were collected. A matching SHA-256 hash supports the first claim, not the second: a compromised application can produce a false record that remains unchanged forever.

User satisfaction is a user's judgment of an application experience, not a direct measurement of model accuracy. Specify the target users, eligible tasks, and observation point. A rating after one answer measures something different from a rating after task completion. Ask one neutral question, such as “How satisfied were you with this interaction?” Do not combine satisfaction, accuracy, and speed in one question.

Begin with prepared foundation model (FM) telemetry at a declared grain: here, one row per application request. Each row contains total inference cost across its attempts, end-to-end latency, model family and version, business unit, and a numeric success flag. Reduce business outcomes to this same grain before joining them; joining several outcome events to one request would duplicate its cost. Define what success means and when an outcome becomes final.

Trajectory evaluation checks how an agent executes a task, not just whether its final answer sounds correct. Its evidence includes tool selections, arguments, returned observations, retries, and termination decisions. A refund confirmation can be accurate while the execution still violates a required authorization check. Conversely, an appropriate tool call can fail because the external service is unavailable.

Nondeterminism is variation across runs with the same apparent inputs. Diagnosis starts by defining the boundary being repeated: the model invocation or the whole application. Repeating a user's question through live retrieval does not hold the model's input fixed. Different document ordering, conversation history, tool responses, or injected dates can change the answer without demonstrating generation variability.

Differential testing compares the same inputs across a baseline and a candidate. A difference is a lead, not a verdict: both responses can be acceptable, and agreement can still be wrong. A versioned golden dataset supplies an independent reference: expected facts, supporting evidence, acceptable behaviors, and failure conditions—not necessarily one exact answer.

Answer relevance asks whether a response addresses the actual request, including its context and constraints. Factual accuracy asks whether its verifiable claims are correct. These are different judgments: truth does not establish usefulness to this request, and directly addressing a request does not establish truth.

A request-conditioned rubric defines success for the actual request, not for a generally informative answer. Before inspecting candidate answers, extract the intended outcome, required subparts, relevant conversation context, and explicit constraints. Express these as observable obligations, each traceable to the request or history.

Evaluate atomic claims: independently checkable assertions that retain their conditions, version, Region, and date. “The feature is available in Region X and enabled by default” contains two claims. Fix decomposition and deduplication rules before comparing systems; otherwise splitting true statements or repeating them can inflate a claim-based score.

Fluency measures how naturally and clearly language is expressed. Factual accuracy measures whether its verifiable claims are true. Neither establishes the other: polished prose can state a falsehood, and awkward prose can express a correct fact.

Fluency measures how naturally and clearly language is expressed for its intended reader. Define the audience, language, expected register, and response format before scoring. Assess grammar, phrasing, sentence connections, and distracting repetition. Do not reward formality or length by default, and keep factual accuracy and relevance as separate scores.

A regression-based release gate asks whether a candidate preserves required behavior relative to an approved, versioned baseline. The candidate is the whole application configuration: model, prompt, retrieval settings, tools, and guardrails. A model identifier alone does not identify what was evaluated.

A canary exposes a limited production cohort to a candidate that has passed offline screening. Version the complete configuration: model identifier, prompt, retrieval settings and index version, inference parameters, guardrails, and tool policy. Record the assigned version with evaluation events. Pin conversations to a version; changing configurations midway confounds both behavior and attribution.

User feedback is a signal, not a representative sample. A thumbs-down records dissatisfaction among people who chose to respond; it does not establish factual error or the failure rate across all requests. Capture a response identifier, an optional reason, and enough protected context to investigate. Keep missing feedback distinct from positive feedback, and remove unnecessary sensitive information before review.

A user rating is an observation about user experience, not automatically a label of response correctness. A fluent but false answer can receive a positive rating; an accurate refusal can receive a negative one. Keep satisfaction, factual correctness, relevance, and safety as separate dimensions rather than averaging them into an unexplained quality score.

A custom Ground Truth labeling job turns prompt–response records into human labels through three contracts: a pre-human-task Lambda prepares display data, a worker template collects rubric fields, and an annotation-consolidation Lambda produces the final label. Ground Truth routes these exchanges; the two functions do not call each other.

A quality gate turns evaluation evidence into permission to proceed. A completed evaluation job or a successful deployment is not permission. Define the gate policy before inspecting candidate results: required measures, risk categories, acceptance rules, evidence requirements, and which failures block promotion. Treat changes to prompts, retrieval, knowledge content, guardrails, inference settings, or the foundation model as potential behavioral changes.

Subgroup analysis estimates quality and harm within meaningful populations rather than treating the overall score as sufficient. Start with affected users, tasks, and plausible harms. Predefine important groups and justified intersections, such as language × accessibility need: acceptable scores for each attribute separately can hide poor results at their intersection. Distinguish user attributes from attributes of people mentioned in generated content. Collect sensitive attributes only under appropriate legal, privacy, and access controls; do not silently infer them from text.

Precision@k asks how much of the top k is relevant. Recall@k asks how much of the relevant set appears in the top k. Let r be the number of relevant items retrieved, and R the total number of relevant items for the query: Precision@k = r/k and Recall@k = r/R.

Precision@k counts relevant items in the first k positions and divides by k. Its validity depends on a stable relevance rubric, not on the retriever’s similarity scores. Define what evidence must support the query and which applicability constraints matter. For credential-rotation instructions, those constraints can include database engine, authentication method, and environment.

Recall@k measures how much labeled relevant evidence appears in the first k results: |relevant ∩ top-k| / |relevant|. If five passages are relevant and three appear in the first ten results, Recall@10 is 3/5 = 0.60. The denominator is five, not ten. With five relevant items and only three result slots, the maximum possible passage-level Recall@3 is 3/5.

Amazon Kendra has been in maintenance since June 30, 2026. It is closed to new customers; existing customers can continue using it. Do not select it for new workloads. AWS provides a migration guide: https://docs.aws.amazon.com/kendra/latest/dg/kendra-availability-change.html. This lesson explains retrieval controls for existing deployments.

Tool-call effectiveness asks whether the agent chose appropriate operations, supplied correct inputs, and achieved the required outcome without avoidable work. A successful API response proves only that an operation executed, not that it was the right operation. Conversely, a dependency outage does not prove that the agent chose badly.

A foundation-model leaderboard is meaningful only when its rows answer the same question. Define a comparison contract: the task, dataset snapshot and split, eligible examples, prompt template, retrieval snapshot when used, decoding settings, scoring rubric, evaluator version, and aggregation rules. For a model-only comparison, hold these fixed and declare the model package as the candidate variable. A prompt comparison deliberately varies the prompt instead. Record that scope explicitly; changing both model and prompt supports a claim about the combined system, not an isolated model improvement.

A confidence interval describes sampling uncertainty around an estimate, not variation among individual answers. A 95% procedure has approximately 95% coverage across repeated equivalent samples when its assumptions hold. It does not assign a 95% probability to the fixed parameter being inside this particular interval. Nor is it a prediction interval, a Bayesian credible interval, or a model-generated confidence score.

A reproducible report starts with a comparison contract: dataset snapshot, eligible examples, model versions, prompt and inference settings, evaluator version, metric definitions, and failure policy. Store one record per example, model, repetition, and metric, with stable identifiers. Deduplicate retries without discarding intentional repetitions. Preserve errors and missing judgments; silently dropping them can reward a model that fails on difficult inputs.

An outcome-based regression suite checks whether a complete synthetic user journey still achieves its intended result after a foundation model update. Enter through the application's real test interface, preserve conversation state across turns, and verify the final response, stored state, and external effects. For an address-change journey, saying “updated” is insufficient: the intended synthetic order must contain the authorized new address, with any required confirmation recorded.

CodeDeploy shifts a Lambda alias between published function versions. An integration update can change prompts, model selection, response parsing, or retrieval behavior without changing the application's API. A successful invocation therefore does not prove that the new version produces acceptable answers.

A useful chunk preserves an evidence unit: the passages needed to support a claim, including its conditions, exceptions, and definitions. Start troubleshooting with a question and its required source passages, not with a preferred chunk size. Trace those passages through extracted text, generated chunks, indexed content, retrieved results, and the final assembled prompt. The first representation that loses necessary evidence identifies the stage to investigate.

A model integration has several contracts: the service operation's request envelope, the selected model's supported inputs, the service response structure, and any application-defined structure inside generated content. A payload can satisfy one contract and violate another. Successful JSON parsing establishes syntax, not conformance to any of these contracts.

A prompt contract regression test checks whether a maintained prompt still produces the structures its consumer expects. Test two boundaries separately: the rendered prompt after fixture inputs are substituted, and the generated structure returned by the model. A template can look correct while an optional branch drops a message or serialization turns an object into a string.