Transfer learning starts from an existing model rather than randomly initialized parameters. During pretraining, a model learns representations such as visual patterns, language structure, or relationships between concepts. Those representations can provide a useful starting point for another task or domain. For example, a language model trained on broad text can be adapted to classify support requests, while an image model trained on general photographs can provide features for identifying manufacturing defects.
Transfer learning is a broad approach, not a single training algorithm. One implementation freezes a pretrained model and uses its outputs as features for a newly trained classifier. Another fine-tunes some or all of the pretrained model's parameters on target data. Parameter-efficient fine-tuning methods, such as low-rank adaptation, train a relatively small set of additional parameters while keeping the original model weights frozen. These approaches differ in training cost, memory requirements, deployment complexity, and ability to change model behavior.
For foundation models, several adaptation methods overlap with transfer learning but describe different aspects of training. Instruction tuning trains a model on instruction-and-response examples so it learns how to follow requests. Domain adaptation specializes a model for a field such as insurance or engineering and can use different training objectives. Continued pretraining, also called continuous pre-training, resumes the model's pretraining objective on additional data, often unlabeled domain text. Instruction tuning and continued pretraining can both transfer capabilities from a pretrained model; they are not mutually exclusive alternatives to transfer learning.
Consider a company building an assistant for equipment maintenance. Continued pretraining on authorized technical documents may help the model learn specialized terminology. Subsequent instruction tuning on reviewed question-and-answer examples can teach the desired response structure and task behavior. These stages are optional design choices, not mandatory steps for every application. If the main requirement is answering questions using frequently updated manuals, retrieval-augmented generation may be a better starting point because it supplies documents at inference time without changing model parameters.
In the cloud, engineers typically select a supported pretrained model, prepare and version training data, run an adaptation job, evaluate the resulting model, and deploy it for inference. Managed customization services simplify supported workflows, while managed training platforms offer greater control over frameworks, training code, and adaptation methods. Check model licensing, customization support, regional availability, data-handling requirements, and deployment options before committing to a model.
Evaluation should compare the adapted model against the original model and a simpler baseline, using held-out examples representative of production traffic. Measure task accuracy, instruction following, safety, latency, and inference cost as appropriate. Keep near-duplicate examples and related records from leaking across training and evaluation datasets. Use modest learning rates, checkpoints, and early stopping where supported to limit overfitting and preserve useful pretrained capabilities.
Transfer learning usually requires less compute and target data than comparable training from scratch, but it does not guarantee better results. Negative transfer occurs when the source model's knowledge is poorly suited to the target problem. Excessive adaptation can cause catastrophic forgetting, reducing previously useful capabilities. Fine-tuning also does not guarantee factual accuracy or reliable storage of changing business facts. Prompting and retrieval alone are not parameter-training methods, whereas transfer learning can include training a new prediction component on frozen pretrained representations without updating the base model.
Amazon SageMaker