Model interpretability concerns whether people can understand the relationships between a model's inputs, learned behavior, and outputs. For example, an engineer investigating a loan-risk prediction might need to understand whether income, repayment history, or a suspicious proxy feature influenced the result. The required level of understanding depends on the audience and the consequences of an incorrect decision.
Transparency and explainability are related but distinct. An intrinsically transparent model exposes a decision process that people can inspect directly, such as a small decision tree or a linear model with understandable features. An explainable system provides information that helps people understand its behavior, potentially through techniques applied after training. A complex neural network can have useful explanations without becoming fully transparent. Conversely, publishing model architecture or weights does not automatically make its behavior understandable.
Interpretability can be global or local. Global interpretation describes overall behavior, such as which inputs generally influence predictions. Local interpretation addresses a particular result, such as why one transaction received a high fraud-risk score. A global feature-importance chart cannot, by itself, explain every individual prediction. Engineers should also check whether an explanation refers to original business concepts or to transformed features that require additional interpretation.
In cloud deployments, teams can run explanation jobs during model evaluation, generate explanations for selected inference requests, and retain reports alongside model versions. Feature-attribution methods such as SHAP estimate how features contribute to a prediction relative to a reference or background dataset. These results depend on methodological assumptions, reference data, and feature relationships. They describe model behavior, not proof that changing a feature would cause a particular real-world outcome.
Generative AI introduces additional difficulties. Large language models have complex internal representations, and their generated text does not provide a reliable account of the internal computation that produced an answer. Asking a model to explain itself can produce a plausible but unfaithful rationale. Retrieval citations can help users inspect supporting documents, but citations do not reveal the model's internal reasoning or guarantee that its claims correctly reflect those documents.
Limited interpretability is a disadvantage of generative AI, separate from hallucinations, inaccuracy, and nondeterminism. A hallucination presents invented or unsupported material as credible content; inaccuracy means an output is wrong; nondeterminism means repeated requests can produce different outputs. An opaque model makes these problems harder to investigate, but opacity does not itself mean every answer is wrong. Likewise, a repeatable answer is not necessarily correct or interpretable.
Practical controls include choosing an understandable baseline model, testing explanations with domain experts, documenting intended use and limitations, and preserving model versions and relevant inference context. For a support assistant, record the retrieved documents, prompt configuration, and response so reviewers can investigate unsupported claims. These records improve traceability without exposing the model's complete internal decision process. Protect sensitive information in both logs and explanation outputs.
Interpretability involves trade-offs rather than a universal preference for simpler models. A compact model may be easier to review but insufficient for a complex task, while post-hoc explanations can add computational cost and latency. High-risk applications may justify a less complex model, stronger explanation validation, or human review. Select the approach by comparing predictive performance, explanation fidelity, operational cost, and the understanding required by stakeholders.