Data Integrity

Data integrity concerns whether data retains its intended meaning and state as it is collected, transferred, stored, processed, and deleted. It covers accidental damage, such as truncated uploads or incorrect transformations, and deliberate manipulation, such as unauthorized changes to training labels. Integrity does not mean data must never change; it means changes must be authorized, valid, and traceable.

In secure data engineering, integrity and data quality overlap but answer different questions. Quality assessment evaluates whether data is fit for its intended use through checks for accuracy, completeness, duplicates, valid ranges, and freshness. Integrity controls also establish whether data has changed unexpectedly and whether approved processing rules were followed. A perfectly preserved dataset can still contain incorrect measurements or biased samples, so successful integrity checks do not establish that an AI model will be accurate or fair.

Apply controls at ingestion before data reaches trusted datasets. Validate schemas, required fields, record counts, identifiers, and business rules, and quarantine records that fail. For example, an AI training pipeline can reject unknown label values and compare incoming record counts with a producer's manifest. Use idempotent processing and deduplication to prevent retried ingestion jobs from silently adding duplicate examples.

Protect data during transfer with properly configured TLS and verify stored or transferred objects using checksums. A cryptographic hash can detect changes when compared with a trusted expected value, but a hash stored beside a file is insufficient against an attacker who can replace both. Message authentication codes or digital signatures can provide stronger evidence of authenticity when verification keys and signing permissions are protected. Encryption at rest primarily protects confidentiality; it does not, by itself, prove that an authorized application wrote correct data.

Restrict write and delete permissions using least privilege, separate ingestion and approval roles, and prevent training workloads from modifying their own approved inputs. Version datasets, transformation code, labels, and configuration together so engineers can reproduce a training run. Retain provenance that identifies the source, processing steps, approvals, and exact dataset version. These controls help investigate data poisoning, although malicious examples can still pass schema and checksum checks and require additional source validation and content review.

Use database transactions and constraints where records must remain logically consistent. For example, updating an observation and its associated label should not leave only one update committed. For object-based datasets, publish a validated manifest or approved dataset version only after all required objects are available. Versioning and appropriate retention controls support recovery and preservation, while audit logs provide evidence of who changed data and when.

Privacy-enhancing techniques must preserve the properties required by downstream processing. Tokenization can preserve record linkage when consistently configured, while masking or differential privacy can intentionally remove detail or introduce noise. Document these transformations and validate their outputs rather than treating every difference as corruption. Balance stronger controls against storage costs, validation latency, operational complexity, and deletion obligations. Backups, immutable storage, and audit logs are complementary controls, not substitutes for validation or tested recovery procedures.