How to Evaluate an AI SOC Tool: A Pilot That Measures Misses, Review Load and Autonomy

A pilot of an AI SOC tool proves something only if the tool's verdicts are hidden from the people judging it, misses are counted with an error bound, and time saved is always reported next to the true positives the tool closed unseen. This guide gives a SOC lead or CISO a 90-day plan for that, from a readiness gate to exit criteria written before day one.

It is for a security team at a US company with a few analysts or an MDR partner that has been offered an AI SOC (also sold as an agentic or autonomous SOC) trial. Most of the pilot guides we found come from vendors, who have a stake in how the trial ends. Everything below is our own practice and analysis unless a source is linked, so change the numbers to fit your team.

Are we ready to pilot?

Not if the tool would be working from missing telemetry. A pilot on logs that were never collected measures your gaps, not the product. Pass four checks first: log sources flowing with their health monitored, a usable asset inventory, a working alert pipeline, and one named pilot owner.

The second check is the gap behind a post on r/cybersecurity about vendor pitches to executives: "we still don't even have a functioning asset inventory". NIST SP 800-61 Rev. 3 lists current, automatically updated inventories of hardware as a recommendation.

For log sources, the priority logs guidance from the Australian Cyber Security Centre and partners including CISA says higher-priority sources should go into a new SIEM first and their health be checked regularly. It loosely orders them, starting with EDR, network devices, domain controllers and Active Directory, Windows endpoints, virtualization, operational technology, cloud, containers and databases. The companion practitioner guidance on SIEM and SOAR says SOAR platforms are "usually not suitable" for environments without a SIEM, with a newly established SIEM, or without an experienced security team. That is about SOAR; applying it to AI SOC tools is our analysis.

"Ready" means working, not perfect. Fix a gap if the tool needs that data to judge the alerts you plan to test; list the gap and move on if it does not.

What baseline do we capture before the pilot?

Capture two to four weeks of how your team handles alerts today, in two parts. Volume and timing come from the SIEM and ticketing system. Accuracy cannot: a closed ticket records what an analyst decided, not whether the decision was right.

From the tickets, pull alerts per day by detection rule, minutes per alert from assignment to closure, time to escalate, and the share closed as benign. Do not take these from a vendor dashboard.

For accuracy, re-adjudicate a random sample of about 100 past closures (our practice), spread across alert classes. A senior analyst, or a second analyst who did not handle the alert, works each one without seeing the original closure and records a verdict. The gap between the two is your baseline error. In the vendor-run phishing trial described below, unassisted analysts reached precision of 0.42 and recall of 0.79 against the study's ground truth, so assume your own closures are imperfect. A tool that agrees with your analysts has shown agreement, not correctness.

Volume alone also misleads. The 2026 SANS SOC Survey summary says incidents handled has been the top reported SOC metric for ten years, cited by 70% of respondents, and "measures volume of activity, not whether the right threats were caught or caught early."

How do we build a test set the vendor has not seen?

Build it from your own history plus controlled emulation, and keep it from the vendor until the pilot ends. Replay confirmed true positives from past incidents, then run emulated techniques on approved hosts. The vendor must not see the list: MITRE's own evaluations apply a similar principle: the adversary focus is announced, but with "No pre-notification of specific tools or techniques".

Pick techniques from your threat model in MITRE ATT&CK. The SIEM and SOAR practitioner guidance, released by CISA with ASD's ACSC, names LSASS dumps, Golden Ticket, living off the land and DCSync as key areas to test. Our starting list, weighted toward identity and lateral movement:

• T1003.001 LSASS Memory, T1003.006 DCSync, T1558.001 Golden Ticket
• T1078 Valid Accounts, T1021 Remote Services, T1046 Network Service Discovery
• T1071 Application Layer Protocol, T1567 Exfiltration Over Web Service
• T1566 Phishing, T1486 Data Encrypted for Impact

Open libraries cover emulation: the Center for Threat-Informed Defense Adversary Emulation Library, Atomic Red Team and Caldera. Recorded attack data is in the OTRF Security Datasets and Splunk attack_data. The vendor may have trained or tuned on public sets, so treat them as filler, not as the blind set. Run nothing in production without written approval and a deconfliction contact, because a seeded test can trip real response. To turn plan assumptions into detection drills, see tabletop exercises.

Four design risks:

1. Hindsight flatters the tool. Indicators from a past incident now sit on reputation lists. Prefer incidents with stale indicators, or score behavior-only cases separately.
2. Replay can understate the tool. Replayed alerts point at hosts and users whose state has changed, and one vendor guide argues that test environments lack the context a tool uses. That vendor sells the tool and favors trials on real data, so run both replay and live emulation.
3. Thin history. A 500-person company may have few confirmed incidents. Emulation fills the gap.
4. No alert, no test. If an emulated technique raises no alert, that is a detection gap in your stack, not a tool miss. Score the two separately.

Include some seeded alerts whose content was written by an attacker, such as phishing bodies and command lines, since verdicts must hold on that input.

MITRE ATT&CK Evaluations do not replace this test. MITRE tells buyers to use results as one signal alongside proof-of-concept testing in your own environment and lists "Effectiveness in your specific environment" as something the evaluation does not measure. As of November 3, 2026, Enterprise 2026 results are not out; MITRE's FAQ puts public publication in December 2026, and it conflicts with the buyer page about whether per-technique scores are public. Do not plan around a vendor's 2026 numbers.

Why must shadow mode hide the tool's verdicts?

Because analysts who see a "benign" verdict defer to it. In shadow mode the tool judges every alert, but its output goes to a store that the analysts and the auditor cannot read. Your team works the alerts as usual; an independent adjudicator later compares both sets of verdicts against the truth.

The evidence is one vendor-run study: Microsoft's randomized trial of its own phishing triage agent, a preprint on that one task, measured before general availability, with no independent replication we could find. Treat it as a method precedent, not proof of how any product performs. In it, 167 analysts each triaged 25 emails drawn from a corpus of 93, and where the study planted wrong "benign" verdicts, analysts shown them missed the threat 46% of the time, against 17% in the control group. It found no such effect on "malicious" verdicts. The authors read this as a reason to skip reviewing benign verdicts. In our view the same table shows that a human reviewing with the verdict in view is a weak check on false negatives, so the check has to be blind.

The same rule covers the audit. A blind auditor sees neither the tool's verdict nor its written rationale.

What do we measure, and how?

Measure misses, review load and autonomy with eight metrics, defined before the pilot starts.

Never report time saved alone. In the trial, the "resolve-benign" protocol, which spares humans from reviewing what the agent called benign, accounted for 79-84% of the productivity gain in the paper's text (78-87% across its table's scenarios). In our reading, that time saving was bought by not looking. Put minutes saved and auto-close misses in the same row of the report.

Report hits and misses per technique, and a rate only for the pool. With no misses in n runs, the one-sided 95% upper bound on the true miss rate is 1 - 0.05^(1/n); the shortcut of 3/n, known as the rule of three, is close from about 30 runs.

Five clean runs of one technique tell you almost nothing. A pooled figure over 30 or more seeded events can say "no misses, true rate below about 10%." Say that, not "zero."

Do not headline accuracy. At 1% truly malicious (an illustration, not a measured rate) a tool that closes everything scores 99%. A tool with 90% recall and a 1% false positive rate scores 98.9%, lower, while closing 10 real attacks per 10,000 alerts. Axelsson's peer-reviewed paper on the base-rate fallacy makes the underlying point that the false alarm rate limits intrusion detection. Ask for recall on known-bad events and precision on what the tool escalates.

How do we audit what the tool closes on its own?

Sample the alerts the tool closed, or in shadow mode would have closed, have the blind auditor re-judge them, and treat one confirmed miss as a stop signal for that alert class. An auditor can miss things too (the trial's unassisted analysts reached recall of 0.79), so slip some planted true positives into the sample and track the auditor catch rate.

We found no primary source with an overturn-rate cutoff, so the stop rule below runs on confirmed misses. The procedure, with owners:

1. Weekly, SOC lead: group auto-closed alerts by class (rule or alert type). Sample at random within each class.
2. Sample size: to claim fewer than 1% of a class's closures were wrong at 95% confidence takes about 300 audited closures with none wrong. If a class closes fewer than 300 alerts in the window, audit all of them.
3. Auditor, within 3 business days: judge each sampled alert blind, with planted true positives mixed in at a rate the auditor does not know.
4. Stop rule: any confirmed true positive closed by the tool suspends autonomy for that class until the root cause is fixed and the class is re-tested. The overturn rate is a secondary signal only: with the verdict in view, it can look low simply because analysts defer to the tool.
5. Verify: the pilot owner checks that each suspended class shows a ticket, a root cause and a re-test date.

The agentic AI guidance from six national agencies, co-authored by CISA and NSA among others, gives the principle behind the stop rule: use continuous evaluation to decide when to expand scope or roll back autonomy after failures.

What may the tool do without a human?

On its own, only enrichment, tagging, linking alerts and opening tickets. Closing as benign, containment and account actions start behind approval; deletion and firewall changes stay off. The placements are our judgment, to be confirmed by the owner of containment decisions in your IR plan. The six agencies' guidance says the decision about when human approval is needed belongs to system designers or operators, not the AI system, and that actions should be classified by impact, likelihood and reversibility.

Sources pull in different directions on isolation. NIST SP 800-61 Rev. 3 tells organizations to consider configuring technologies to contain automatically, including moving a compromised endpoint to an isolated network. The six agencies, who do not name host isolation, say to insert human approval checkpoints for actions where an error is costly, and that organizations should only use agentic AI for low-risk and non-sensitive tasks. A tool that reads your security logs and can act on production does not obviously meet that last test. The SIEM and SOAR guide also warns that a SOAR platform that is not properly configured may misidentify normal behavior as an incident and act to isolate it.

NIST is speaking of configured technology and the agencies of AI agents. Our view is approval by default, with named classes pre-authorized in the IR plan once the pilot has proved them.

What are the phases, and who owns them?

Five phases over about 90 days, with a go or no-go check at each boundary. Negotiate a longer trial if the vendor offers 30 days: a short one tends to end before the audit has enough closures to mean anything.

1. Readiness, weeks 1-2, SOC lead. Run the gate checks. Sign the pilot plan, naming the person who may suspend the tool. Verify: all four checks pass or are waived in writing.
2. Baseline, weeks 1-4, SOC lead and a senior analyst. Pull ticket data and re-adjudicate the sample. Build the test set in parallel with the detection engineer. Verify: baseline numbers are stored before the tool sees an alert.
3. Shadow mode, weeks 5-8, pilot owner. Hidden verdicts, seeded events at planned intervals, weekly audit. The independent adjudicator is a senior analyst or detection engineer who does not work the queue during the pilot; a small team uses a peer from another team or the MDR partner's reviewer (our practice). Verify: analysts confirm they cannot see tool output.
4. Limited autonomy, weeks 9-12, pilot owner with the IR lead. Allow named, low-risk alert classes only, with the stop rule active. The agencies recommend phased deployment with progressively increasing access and autonomy. Verify: the action allowlist is enforced by configuration, tested once by attempting a blocked action.
5. Decision, week 13, CISO. Apply the exit criteria below and write the result down.

What exit criteria should we set?

Write three outcomes and their thresholds before day one: buy, buy with limits, stop. NIST's AI RMF says test sets, metrics and tools should be documented, and that assessors who did not build the system should take part; the vendor does not grade its own pilot.

What do we ask the vendor in writing?

Ask for the controls the pilot depends on, and put the answers in the contract:

• A per-verdict evidence trail: what data it read and why it decided as it did.
• An export of every decision and its evidence, in a format you keep after termination.
• An action allowlist you set, a kill switch, and a named way to deactivate the tool. NIST's AI RMF calls for the ability to supersede, disengage or deactivate systems that perform outside intended use.
• Behavior when uncertain: the agencies recommend that systems stop and escalate to human reviewers in uncertain scenarios.
• Behavior when a data source goes silent: whether it says so, or closes alerts as if nothing happened.
• A right to re-test after model or version changes, and no training on your data without opt-in.

On the data the tool ingests (alert content, log contents, retention, model training), ask the same questions as for any vendor that holds sensitive security telemetry.

Build your pilot plan

Fill this in before the trial starts. Your answers stay in your browser.

What to do this week

1. SOC lead: run the four gate checks and list failures.
2. CISO: name the pilot owner and ask the vendor for the written answers above before the trial.
3. Detection engineer: choose technique IDs and approvals for the test set.
4. SOC lead: start the ticket pull and the re-adjudicated sample.