Semi-Structured Data

Semi-structured data has recognizable organization but does not require the uniform rows and columns typically associated with structured relational data. Keys, tags, nested objects, and arrays describe relationships within individual records. Common examples include JSON API responses, XML documents, and application logs that contain named fields. Two records can represent the same kind of event while carrying different optional attributes.

Consider an online store emitting purchase events. Each event might include an event identifier, timestamp, customer identifier, and an array of purchased items. One event might also contain a coupon code, while another includes delivery instructions. This is semi-structured because the records have meaningful fields and hierarchy without requiring every event to contain an identical set of attributes. By contrast, a relational purchase table usually defines consistent columns and types. Free-form customer comments, photographs, and audio recordings are generally treated as unstructured content for analytics and AI, even though their files have technical formats.

In cloud architectures, engineers commonly ingest these records through APIs, message streams, or batch file uploads. Object storage can retain the original payloads for replay and auditing, while document or key-value databases can support application access. Processing jobs parse records, validate important fields, normalize timestamps, resolve inconsistent types, and extract attributes for analytics. Frequently queried data can be transformed into columnar formats such as Parquet to reduce scanning costs and improve analytical performance.

Schema-on-read means applying an interpretation when reading data rather than enforcing all structure before storing it. It is useful for retaining evolving source payloads, but it does not mean there is no schema. Producers and consumers still need agreements about identifiers, field meanings, types, and compatibility. Schema validation can also occur at ingestion. A JSON document may be semi-structured yet governed by a strict schema; the format alone does not determine how flexible a dataset actually is.

The three data characteristics of volume, velocity, and variety describe different engineering pressures. Volume concerns how much data must be stored and processed. Velocity concerns how quickly data arrives and how soon it must be processed. Variety concerns differences in formats and representations, including structured, semi-structured, and unstructured data. A stream of JSON events can have high velocity and volume, while changing event fields add variety. These dimensions affect buffering, partitioning, processing capacity, and validation design.

For AI and machine learning, semi-structured records often connect raw content to labels, metadata, and business context. A JSON training manifest might reference an image in object storage and provide its class label, capture time, and provenance. The manifest is semi-structured; the referenced image remains unstructured content. Similarly, wrapping free-form text or an audio reference in JSON does not make the underlying content ready for a model. Text may need tokenization, images may need decoding and resizing, and audio may need resampling or feature extraction.

Flexibility shifts work downstream. Missing fields, unexpected nesting, numeric values encoded as strings, and incompatible schema changes can break queries or corrupt training features. Engineers should distinguish absent fields from explicit null values, quarantine invalid records, version incompatible event contracts, and monitor parsing failures. Flattening arrays also requires care because it can multiply rows and distort aggregates. For model preparation, preserve labels and provenance, prevent target leakage, and apply consistent transformations during training and inference. Semi-structured data reduces the need for a rigid source layout, not the need for data quality, governance, or deliberate modeling.