A transformer is a neural network architecture built around attention mechanisms rather than the recurrent processing used by earlier sequence models. Attention lets a model weigh relationships between elements in its input. In a sentence such as “The server rejected the request because it was overloaded,” attention can help represent relationships between the pronoun and surrounding words. These relationships are learned statistical patterns, not evidence that the model understands the sentence as a human would.
For text, processing begins with tokenization. A token can represent a word, part of a word, punctuation, or another unit defined by the tokenizer. Token identifiers are mapped to embeddings: numerical vectors that provide learned representations. Positional information is also incorporated because attention alone does not establish sequence order. Within transformer layers, attention uses learned query, key, and value representations to combine relevant information. Multiple attention heads can capture different relationships, while feed-forward networks, normalization, and residual connections help transform and propagate representations through the model.
There are several major transformer configurations. Encoder-only models commonly support representation and classification tasks. Decoder-only models commonly generate text by predicting the next token using preceding tokens, with causal masking preventing access to future positions. Encoder-decoder models combine input encoding with output generation and are often used for sequence-to-sequence tasks such as translation. Not every transformer is a generative model, and not every generative model uses transformers.
A large language model is a model trained to process and generate language at scale; many modern LLMs use transformers. A foundation model is broadly pretrained and adaptable to multiple downstream tasks, rather than being an architectural category. Foundation models can process language or other modalities. Multimodal transformers can work with representations of text, image patches, audio, or combinations of these inputs. Diffusion describes a generative modeling approach that learns to reverse a noise-adding process; its neural network can use a transformer or another architecture. These terms describe different aspects of a system and are not interchangeable.
In cloud applications, engineers can invoke a managed transformer-based model through an API or deploy model weights on suitable compute infrastructure. For example, a support assistant can retrieve relevant troubleshooting passages, place them alongside a user question in a prompt, and ask a language model to generate an answer. Prompt engineering shapes the instructions, examples, and context supplied at inference time; it does not normally change model weights. Fine-tuning, by contrast, updates model parameters through additional training.
Chunking divides documents into manageable passages for retrieval or processing; it is not the same as tokenization. An embedding model can convert those passages into vectors for similarity search. Retrieval-augmented generation then supplies selected passages to the generator. The retrieval embeddings and the generator’s internal token representations have different roles and need not come from the same model. Retrieval can improve grounding without retraining the generator, but it does not guarantee factual answers.
Transformers offer substantial parallelism during training, but autoregressive text generation usually proceeds token by token. Standard full self-attention has quadratic attention computation with sequence length, so longer contexts can increase resource requirements substantially; optimized implementations and alternative attention patterns can reduce particular costs. Engineers should evaluate context-window limits, output length, latency, throughput, accelerator memory, and pricing. A larger context window does not guarantee that every detail will be used reliably, and fluent output does not guarantee correctness. Production systems still require evaluation, access controls, sensitive-data handling, and appropriate validation of generated results.
Amazon EC2