Many powerful AI models, especially in deep learning, consist of millions or billions of parameters. They deliver results without it being readily apparent to humans how these come about. This is referred to as a black box. Explainable AI aims to open this black box to some extent: for trust, for troubleshooting, and wherever decisions affect people.
How does explainable AI work?
There are two basic approaches:
- Interpretable models: Decision trees, linear models, or rule sets are inherently understandable. For structured data, they are often sufficient.
- Post-hoc explanations: For complex models, explanations are generated afterwards. Methods such as SHAP or LIME show which input features influenced a result and how strongly. For image data, heatmaps highlight the decisive image regions. Counterfactual explanations describe what would have had to change for the result to turn out differently.
For language models, source citations, for example in retrieval-augmented generation, help make answers verifiable. Justifications that a language model formulates itself, by contrast, do not necessarily reflect how the answer actually came about internally.
Explanations are approximations. They can be incomplete or misleading and must themselves be critically examined.
Typical examples
- In quality control, a heatmap shows that a component was rejected because of a scratch.
- In predictive maintenance, it is apparent which sensor values triggered a warning.
- A knowledge assistant refers to the underlying documents with every answer.
- During model development, it turns out that an image model pays attention to the background instead of the actual object, an indication of weaknesses in the training data.
What companies should keep in mind
- Clarify who needs which explanation: the business department, affected persons, auditors, or the development team.
- Prefer simpler, interpretable models if they deliver comparably good results.
- Demand documentation, explanation features, and source citations from vendors.
- Do not treat explanations as proof of correctness, but check them on a sample basis.
- Take into account transparency requirements that the GDPR and the EU AI Act, for example, provide for certain automated decisions and systems.
How it differs
Explainable AI is closely related to the topic of bias but is not the same: explainability describes whether a result can be understood, bias whether it is skewed. Only when it is apparent what a model pays attention to can biases be uncovered in a targeted way.
View AI consulting