Cloud concept · Updated

Continued Pre-Training - Continued pre-training further trains an existing foundation model on additional data using a self-supervised objective to adapt its learned representations and knowledge.

In short

Continued pre-training builds on a foundation model rather than training a new model from scratch, typically using unlabeled documents from a target domain or a newer corpus. It can improve familiarity with specialized language and concepts, but requires careful data preparation, compute budgeting, and evaluation to avoid degrading existing capabilities.

Cloud Concepts Hub →

6 key aspects of Continued Pre-Training

Continued pre-training adapts an existing model through self-supervised training on additional data, with success depending on corpus quality, controlled execution, and regression testing.

01
Reuse of a pretrained foundation model

Training starts from an existing model checkpoint rather than randomly initialized parameters. Confirm that the model license, available weights, and platform capabilities permit the intended adaptation.

02
Self-supervised learning from additional data

Training targets are derived from the data itself, such as predicting the next token in a document. This avoids manual response labeling, but still requires substantial corpus cleaning and quality control.

03
Adaptation of knowledge and representations

Additional training can improve familiarity with specialized vocabulary, writing styles, and conceptual relationships. It does not turn model parameters into a reliable document database.

04
Distinct from instruction tuning and retrieval

Instruction tuning teaches desired response behavior through examples, while retrieval adds external context at inference time. Continued pre-training changes model parameters through further pre-training and can be combined with either approach.

05
Regression-aware evaluation

Evaluate domain improvements alongside general reasoning, instruction following, and safety. Use held-out datasets and downstream tests to detect overfitting, memorization, and catastrophic forgetting.

06
Controlled training lifecycle

Version datasets, configurations, and checkpoints, and budget accelerator usage before starting a job. Repeat training only when measurable benefits justify the cost and operational risk.

Cloud services for Continued Pre-Training

Amazon S3 can hold curated training and evaluation datasets, with access restricted to the identities running the workload. Amazon Bedrock offers a managed customization path when the selected model supports continued pre-training. Amazon SageMaker AI provides an alternative for custom training code and compatible model weights when more control is required. These are alternative training paths rather than mandatory consecutive stages. Across either path, retain dataset versions, evaluate the adapted model against the original, and deploy only after checking domain performance, general capabilities, and safety.

Amazon Bedrock

Amazon Bedrock provides managed continued pre-training for supported foundation models through model customization workflows. Availability depends on the model and Region, so verify support, dataset requirements, and deployment options before choosing this path.

Amazon SageMakerAmazon SageMaker

Amazon SageMaker AI can run custom continued pre-training workloads using compatible model weights, training frameworks, and accelerator instances. It provides greater control over training objectives, distributed execution, checkpointing, and evaluation than a fixed customization interface.

Amazon S3Amazon S3

Amazon S3 stores training corpora, validation datasets, and training artifacts for cloud-based adaptation workflows. Access controls, encryption, and versioning help protect data and preserve the inputs associated with each experiment.

Continued Pre-Training exam questions

Real practice exam questions on Continued Pre-Training, with the correct answer and its explanation. Answer them with every option in the exam simulator.

AIF-C01

A publishing company uses Amazon SageMaker AI to adapt an existing language foundation model by training it on licensed archives using next-token prediction. A licensing correction later identifies a collection that was not authorized for training. Company policy prohibits deploying any model version trained on that collection. The company retains the original foundation model checkpoint, which predates the archive training and meets its licensing requirements. It wants to retain adaptation to the remaining authorized archives. Which action should the company take?

Show answerCORRECT ANSWER

Restart adaptation from the original foundation model checkpoint using only the authorized archives, evaluate the result, and replace the current customized model.

Continued pre-training updates model parameters using the training corpus. Restarting from an acceptable checkpoint and excluding the unauthorized collection produces an adapted model whose training meets the stated policy.

MLA-C02 (beta)

An engineering company has a large collection of unlabeled equipment manuals in Amazon S3. The company wants to update a foundation model's parameters so that the model becomes more familiar with specialized maintenance terminology, rather than teach it a particular question-answering format. The company selects an Amazon Titan Text model in Amazon Bedrock that supports the required customization methods in its AWS Region. Which data preparation and customization approach best meets these requirements?

Show answerCORRECT ANSWER

Remove duplicate and low-quality content, redact sensitive information, and divide the text into records within the model's supported limits. Store a JSONL file in S3 with an "input" field containing text in each record, and use it for continued pre-training.

Continued pre-training updates an existing model using unlabeled domain text. For supported Amazon Titan Text models, the training dataset uses JSONL records containing an "input" field without a corresponding labeled output. Cleaning the corpus and respecting model-specific limits prepares the manuals for this objective.

Practice Continued Pre-Training in a full exam

Questions on Continued Pre-Training in each practice exam, with every answer option and an explanation for each:

Start a free practice exam

What is Continued Pre-Training

2 min read

Continued pre-training resumes the pre-training process of an existing foundation model using additional data. For a generative language model, this commonly means training on documents with a next-token prediction objective. The training targets come from the text itself, so engineers do not need to label each document with a correct answer. The process updates model parameters and can change how the model represents terminology, patterns, and relationships.

A practical example is adapting a general-purpose language model to industrial maintenance. An engineering organization could prepare a corpus of equipment manuals, inspection reports, and technical publications, then continue training a suitable model on that material. The goal is to improve its familiarity with maintenance language and concepts, not simply to make it reproduce individual documents. The organization must have appropriate rights to use the material and remove sensitive information that should not enter model training.

In a cloud implementation, engineers typically place a versioned training corpus in object storage, select a compatible pretrained model, and run a managed customization job or a custom training workload on accelerators. Data preparation includes deduplication, quality filtering, tokenization, and separating training data from validation and evaluation data. Checkpoints, training configuration, dataset versions, and metrics should be retained so that results can be reproduced and compared.

Continued pre-training differs from instruction tuning. Instruction tuning uses examples of instructions and desired responses to teach task behavior, such as answering a question in a required format. Continued pre-training primarily learns from the underlying content and statistical structure of additional data. A workflow may perform continued pre-training first, followed by instruction tuning and safety evaluation. Both are forms of adapting an existing model, while transfer learning is the broader principle of reusing learned capabilities for another setting.

Domain adaptation is a goal, not a synonym for this technique. It can also be achieved through supervised fine-tuning, retrieval-augmented generation, or combinations of approaches. Retrieval supplies relevant documents during inference without updating model parameters. It is often preferable when answers require current facts, document citations, or access-controlled information. Continued pre-training may improve domain fluency, but it does not guarantee accurate recall, eliminate hallucinations, or provide traceable sources.

The main trade-offs are training cost, operational complexity, and potential regression. A large corpus can require substantial accelerator time, and more training does not automatically produce a better model. Excessive specialization can cause catastrophic forgetting, where general capabilities deteriorate. Engineers can mitigate this risk through conservative learning rates, appropriate mixtures of general and domain data, and checkpoint selection based on both domain-specific and general evaluations. For language models, validation loss or perplexity is useful, but downstream task quality and safety must also be tested.

Despite the alternative name continuous pre-training, this is not necessarily an always-running process or automatic learning from user conversations. It is commonly executed as a controlled batch job. Before deployment, compare the adapted model with the original model and simpler alternatives, assess memorization and sensitive-data exposure, and verify that any required instruction-following and safety behavior remains effective.

Learn Continued Pre-Training in lessons

4 short lessons on Continued Pre-Training in 2 certification courses.

Open the lessons