JSON

JSON is a language-independent serialization format used to exchange and persist data. An object contains name-value pairs with double-quoted string names, while an array contains an ordered sequence of values. Values can also be strings, numbers, booleans, or null. For example, {"customer_id":"c42","purchase_total":29.95,"items":["book","pen"]} represents a record with both scalar attributes and a nested collection. JSON object member order should not carry business meaning; array order does matter.

Cloud applications commonly produce JSON through APIs, event streams, and application logs. A data pipeline can preserve these records in object storage before validating and transforming them into machine learning features. JSON is useful when records contain nested structures or optional fields that would be awkward to represent in a flat CSV file. However, structural flexibility does not eliminate the need for a data contract: producers and consumers must agree on field meanings, types, required values, and schema evolution.

Record framing is an important ingestion decision. A conventional JSON document might contain one object or a large array of objects. JSON Lines, commonly called JSONL or newline-delimited JSON, instead places one complete JSON value on each line, usually one object per record. The entire file is not one conventional JSON document. This layout allows compatible readers to process records incrementally and can support parallel processing of uncompressed files. A pretty-printed multiline object is not interchangeable with a single-line record when the reader expects JSON Lines.

Choose the format according to the access pattern. JSON is a practical choice for exchanging complete records and retaining heterogeneous source events. CSV is often simpler for flat tabular data, although quoting, null handling, and type interpretation require agreement. Apache Parquet and ORC are generally better suited to repeated analytical queries that select a small subset of columns from large datasets. Their columnar layouts, encoding, compression, and metadata can reduce data reads compared with scanning textual JSON records.

For example, an ecommerce pipeline might ingest purchase events as newline-delimited JSON, retain the original events for replay, and transform validated records into partitioned Parquet for feature engineering. A daily job that reads only customer identifiers, timestamps, and purchase amounts can then avoid repeatedly parsing large JSON payloads containing unrelated attributes. If a training algorithm requires CSV or another supported format, the preparation pipeline must write that format explicitly; storing JSON does not make it acceptable to every training container.

Compression reduces JSON storage and transfer volume, but it does not turn JSON into a columnar format. Common gzip-compressed files generally cannot be split for parallel reading, so a few very large compressed objects can limit throughput. Balance file sizes and object counts for the chosen reader, and distinguish storage partition pruning from column pruning: date-based object prefixes can reduce which files are read, while JSON still generally requires parsing the selected records.

Strict syntax and data quality checks prevent subtle failures. Standard JSON does not permit comments, trailing commas, NaN, or Infinity. Dates and binary data have no native JSON types and need agreed representations, such as timestamp strings and base64-encoded strings. Missing fields and explicit null values are not necessarily equivalent. Large integers can lose precision in consumers that use floating-point numbers, so identifiers may need string representations. Validate records, quarantine malformed inputs, and check reader-specific behavior for duplicate keys, nested arrays, and changing field types before using the data for training.