Why Explainable AI in Cyber Security Matters
Why Explainable AI in Cyber Security Matters
Shaun Archer
7/6/20267 min read
Why explainable AI is becoming non-negotiable in cybersecurity
For years, security vendors got away with selling AI as a black box. You gave up visibility into how a model reached its verdict, and in return you got detection at a scale no human team could match. Fair trade, or so it seemed.
That trade is starting to fall apart. Analysts are drowning in alerts they can't interpret. Regulators are asking for auditable decision logic. And attackers, who study defences the way anyone studies an opponent, have got quite good at exploiting the gap between "the model flagged it" and nobody being able to say why. Explainable AI, or XAI, is the industry's answer: model outputs that come with a reason attached, so a human can check the reasoning rather than just take the verdict on faith.
It matters more here than in most other applications of machine learning. If a recommendation engine gets something wrong, you get a bad film suggestion. If a security model gets something wrong and can't explain itself, an analyst either rubber-stamps an alert they don't understand or dismisses one they can't justify keeping. Neither is a good outcome, and a black box makes both more likely.
Why explainability actually matters
Start with alert fatigue, because it's the most immediate problem. SOC teams already sit in front of thousands of daily alerts pulled from email, identity, endpoint, and cloud logs. A score of "risk: 87" tells an analyst almost nothing they can act on. Compare that to: this user signed in from a new country, immediately set up a mail forwarding rule, and pulled down forty times their normal volume of files from a finance-tagged SharePoint folder. That second version lets someone triage in seconds instead of piecing the story together from raw logs themselves. Good explanations don't just build confidence in the tool — they buy back the one thing every SOC is short on, which is time.
There's a trust problem underneath that. Analysts who don't understand why a model fired tend to go one of two ways: they either defer to it blindly, which means they stop catching what it misses, or they tune it out after enough unexplained false positives, which means they stop catching what it actually gets right. A detection that can't explain itself has a shelf life. Sooner or later it gets ignored.
Regulation is now pushing in the same direction, and it's worth naming specifically rather than gesturing at vaguely. The EU AI Act requires auditable decision logic for high-risk AI systems, with those obligations phasing in through 2026. The EU's Digital Operational Resilience Act, in force for financial entities since January 2025, doesn't use the words "explainable AI" outright, but its ICT risk management and audit requirements push firms toward the same place — if you can't explain a decision, you can't audit it, and if you can't audit it, you're not compliant. Security teams buying or building AI-driven detection need to be able to answer "why did the system do that" for reasons well beyond a regulator's clipboard, too. Legal and incident response teams will ask the same question after any real incident.
Which gets at the part that's easy to underweight until it happens to you: defensibility. When a detection leads to an account lockout, a blocked wire transfer, or an HR escalation, somebody eventually has to justify that call — to the employee it affected, to legal, maybe to a regulator, possibly in a deposition. "The model said so" does not hold up in any of those conversations. A model that can point to the specific behavioural deviations and connect them to known adversary tradecraft gives the people doing incident response something they can actually stand behind.
And then there's the adversarial angle, which cuts both ways. If a model is a black box to the defenders running it, it's genuinely hard to notice when it's started weighting a spurious feature, or when someone has found a blind spot in it. Explainability isn't only a courtesy to end users — it's also how the engineers maintaining the model catch it drifting before an attacker does.
Where MITRE ATT&CK fits in
Explanations need a shared vocabulary, and in security that vocabulary already exists. MITRE ATT&CK is a knowledge base of adversary tactics and techniques, built from observed real-world intrusions, organised into tactics (the attacker's goal — Initial Access, Exfiltration, and so on), techniques (how they get there — Phishing, OS Credential Dumping), and increasingly granular sub-techniques. The current Enterprise release, v19 from April 2026, runs to 15 tactics, 222 techniques, and 475 sub-techniques, and it recently split the old "Defense Evasion" tactic into separate Stealth and Defense Impairment tactics — a sign of how much more precisely the field now wants to talk about adversary behaviour.
ATT&CK gives a model's output somewhere better to land than a bare probability score. Instead of "this is anomalous," a well-grounded pipeline can say something like "this looks like T1098, Account Manipulation, following T1078, Valid Accounts — a sequence that shows up regularly in business email compromise." That single sentence does a few things at once. It puts the alert in language analysts already speak. It lets a team measure its own coverage against known adversary behaviour, which surfaces the gaps in what it can actually detect and explain. And it grounds the explanation in a taxonomy that means the same thing everywhere, instead of a vendor's private labelling scheme that only makes sense inside one product.
In practice that means ATT&CK shouldn't be something you bolt on as documentation after a detection ships. It should shape feature engineering from the start, organise how detection logic is built and reviewed, and tag every high-confidence output with the tactic or technique it most resembles.
What "not normal" actually looks like
None of this works without a clear idea of what abnormal means, and "abnormal" is always a comparison, never an absolute. A model has to be able to point at the baseline it's measuring against, and then at the specific thing that deviated from it. In practice, a handful of signals tend to separate genuine variance from behaviour worth escalating:
Impossible or improbable travel is the classic one — authentication from two locations far enough apart that no flight could explain it, or a first-ever login from a country the user, their team, and the wider organisation have simply never touched before. Off-baseline timing and volume matter too: someone who works a predictable 9-to-5 suddenly active at 3am, or moving data at many multiples of what they normally do.
Mailbox and identity manipulation deserves its own mention because it's such a reliable signal of compromise — a newly created inbox rule that quietly auto-forwards or deletes mail, an unexpected MFA re-registration, or an OAuth consent grant that quietly expands what an app can touch. Lateral movement is another: an account that has only ever touched marketing shares suddenly authenticating against finance systems or a source-code repository it has no history with.
Peer-group deviation is subtler but just as useful — behaviour that looks unremarkable in isolation but is clearly off relative to the people actually doing that job (an individual contributor querying the same systems a database admin would). Communication pattern shifts matter for the same reason: a vendor or executive account that abruptly changes its tone, urgency, or starts making payment requests unlike anything in its history is a strong tell for BEC or vendor impersonation. Privilege creep — self-granted permissions, or access requests that route around the usual approval chain — and MFA fatigue, where someone gets hit with a burst of push notifications or one-time codes in a short window, round out the list.
None of these is meaningful as a standalone rule. Each one is completely legitimate in the right context. What actually makes something abnormal is a documented deviation from an established baseline, and surfacing that deviation in plain language — not just flagging it — is the whole job of an explainable model.
Getting it built
The baseline comes before the model, not after. Explainability is only as good as what it's being measured against, so the real work starts with per-identity and per-peer-group baselines: typical login geography, typical data volumes, typical send patterns. That baseline needs to be inspectable on its own terms — an analyst should be able to see "here's what normal looks like for this person" as easily as "here's what changed."
Not every part of the pipeline needs a complex model. Decision trees, rule-based scoring, and logistic regression are interpretable by construction and handle a lot of well-understood attack patterns just fine. Where deep learning or gradient-boosted models genuinely earn their complexity, pair them with post-hoc explanation methods — SHAP for feature-level attribution with solid theoretical backing, or LIME when SHAP's computational cost is too much at scale. Neither of these fixes a bad baseline; they only explain what the model weighted, not whether the underlying comparison was sound to begin with.
The harder and more overlooked part of implementation is translation. A SHAP output ranking "login_country_entropy: 0.42" as the top feature is not something an analyst can use. Turning that into a sentence they can read in two seconds — ideally one that references the ATT&CK technique the pattern resembles — is where most of the actual engineering effort should go.
Everything needs to be logged for auditability: the explanation, the model version, the features behind the call, kept for as long as incident response and regulatory retention require. That's the difference between a black box and something you can defend after the fact if you're ever asked to.
Feedback needs somewhere to go, too. When an analyst overrides a verdict, that correction should flow back into the model or the baseline — and the reason the model was wrong is usually more useful than the correction itself, since it's often a sign the baseline had gone stale or a feature was miscalibrated. Finally, test the explanations under adversarial pressure, not just for accuracy. A model that shows its reasoning is showing that reasoning to attackers too, so it's worth deliberately checking whether small behavioural tweaks can flip a verdict without changing the underlying intent.
None of this makes detection AI simpler. It makes it something a security team can actually stand behind — in a SOC handoff, in an audit, in front of a regulator, or in court. Grounding explanations in a shared framework like ATT&CK, and being precise about what "abnormal" means relative to a real baseline, is what separates a system that produces alerts from one that produces understanding.