Before extending an AI system’s tools or authority, leaders need to know which evaluation results still support deployment claims, and which remain untested.

Executive takeaways

  1. Evaluate the deployed AI system, not the model in isolation. A change in connected tools, permissions, data or instructions can alter what earlier evidence supports.
  2. Preserve an examinable link between the test and the claim. Records should show what was tested, what changed, what behaviour was observed and what remains unknown.
  3. Keep evidence and decisions separate. Operators hold engineering and release authority; independent examination has a bounded scope; insurers decide whether to accept, price or exclude an exposure under their own terms.

The assurance question begins where the test ends

Illustrative example — synthetic, not a reported deployment or an examined system. An energy operator evaluates an AI assistant for fault diagnosis using simulated telemetry. Before deployment, the operator (or its supplier) changes the data source and instructions and adds a tool that drafts operational instructions for human approval. The original test did not exercise that workflow. If the operator relied on the earlier result for the new function, it could authorise a process whose behaviour nobody had evaluated, and an inappropriate instruction could contribute to an operator error or interruption. This is a plausible scenario, not an observed incident or demonstrated consequence. Whether any technical error caused damage, financial loss, liability or an insurance response would need separate evidence.

This is an AI assurance problem that sits alongside, and does not replace, industrial engineering and safety practice. The question is whether a stated claim about AI controls or behaviour can still be supported when the system boundary changes. An organisation may have strong operational technology (OT) engineering practices, meaning the engineering of systems that monitor and control physical processes, and still lack a traceable account of which AI evaluation results apply to its deployed system.

ODA3 Institute’s assessment is that a release or procurement decision should be able to distinguish the observed test result, the claim being made about deployment and the differences between them. A test result is not a substitute for that comparison, and the comparison alone does not establish that an energy installation is safe to operate.

DOE’s Stormbreaker provides a concrete illustration

On 16 July 2026, the US Department of Energy’s (DOE) Office of Cybersecurity, Energy Security, and Emergency Response (CESER) described Stormbreaker, developed with Lawrence Livermore National Laboratory, as a dynamic testbed (a controlled environment for evaluations) for large language models (LLMs) and agentic AI (AI connected to tools so that it can take actions) in power systems and OT environments [1]. DOE says an AI system’s performance depends on the underlying model, instructions, prompts, skills, tools and target environment together, and that static benchmarks change only one of these variables at a time [1]. It describes configuration transfer, which systematically varies instructions, prompts, tools, skills or the environment, and limit discovery, which makes a scenario progressively harder until failure reveals a capability boundary [1].

DOE says Stormbreaker builds on Mjölnir, described by CESER on 14 April 2026 as a testbed for adversarial evaluation of AI models used in energy applications [2]. Both are programme descriptions. They are not published proof that a particular utility deployment passed an independent assessment.

DOE also says Stormbreaker is intended to support stakeholders interested in safe and effective AI deployment, and names defensive, offensive and adversarial security tasks as uses for testing [1]. DOE does not prescribe ODA3 Institute’s proposed evidence-continuity practice, claim that Stormbreaker authorises plant release, or state that an insurer relies on its results. Applying its configuration-aware perspective to the evidence behind deployment claims is ODA3 Institute’s analysis, not an endorsement or recommendation by DOE.

This is US federal programme material. The analytical principle, that evidence must be interpreted within its tested configuration, can be applied across jurisdictions; regulatory and insurance treatment may differ. Potential cross-sector implication (ODA3 analysis): the same evidence-boundary problem could arise when an enterprise AI agent gains an additional application permission. That is a plausible analogy, not an observed result in the sources reviewed.

Evidence continuity requires more than a successful test

Evidence continuity here means maintaining a traceable relationship between an evaluated AI configuration, observed behaviour and the claim made about its deployed configuration. A claim such as “this AI assistant was evaluated under realistic conditions” is too broad to show what the evaluation supports. A more examinable claim identifies the system version, tool permissions, test inputs, scenario population, observed outputs and the deployment configuration to which the result is applied. Where a supplier ran the test, the operator should request the same records.

ODA3 Institute’s analysis favours a bounded evidence comparison over a universal score or mandatory template. For an individual deployment claim, the decision-maker should be able to inspect four elements:

  • Evaluated boundary: model and application versions; instructions; tools; access permissions; connected data; scenarios and evaluation period.
  • Observed behaviour: successes and failures; tool calls; adverse cases; test coverage and any conditions intentionally excluded.
  • Deployment boundary: changes to interfaces, data, permissions, human review, operational constraints and software dependencies.
  • Reliance limits: which proposition the evidence supports, which assumptions remain untested, and what change would require re-examination.

These are evidence questions, not a certification standard or a prescribed assessment schema. Absence of relevant records means a proposition may remain unknown; it is not evidence that a control failed. Records showing a material mismatch may support a narrower conclusion about the tested-versus-deployed claim without proving that the installation is unsafe.

Decision question Evidence to examine Bounded conclusion it could support Remaining uncertainty
Was the claimed AI behaviour actually observed? Scenario definition, versioned test logs, inputs and outputs What happened in the recorded scenarios Behaviour outside those scenarios
Is the AI application’s authority unchanged? Tool inventories, permission policies, tool-call records Whether evaluated and proposed authority correspond Consequences of new or altered tools
Did the operating context change? Data source and interface comparisons, timing assumptions, configuration records Which conditions differ from testing Effects of unrepresented conditions
Are failure observations preserved? Negative test cases, exceptions and limit-discovery logs Observed failure boundaries under test Failure modes not tested or reproduced
Who decides whether the evidence is sufficient for the proposed use? Named claim owner, acceptance criteria, review record and exceptions An accountable reliance decision and its stated basis Adequacy of criteria for plant engineering or legal purposes

Independent examination has a defined boundary

Management owns the system, its controls and its evidence. The operator’s competent engineering authority remains responsible for plant safety, operating procedures, protection systems and operational release.

An independent examiner may compare the test and deployment configurations, inspect whether stated controls leave records, challenge the coverage of test scenarios and reproduce selected observations in an authorised environment. That requires independence from the underlying claim, a defined commission, relevant competence, conflict-of-interest controls, and agreed rights of access and reliance.

The resulting conclusion must identify the exact claim, configuration, evidence period, observed conditions and exceptions, and what was not examined. Inspecting AI tool permissions is not certification of a plant’s physical protection systems. A desk review cannot substitute for plant engineering tests. Independent examination cannot guarantee future AI behaviour or justify unrestricted control authority.

For this article, no assurance finding is assigned to Stormbreaker, Mjölnir, any supplier or any utility. ODA3 Institute tested no AI-enabled plant, and we reviewed no record of operator or insurer reliance on an ODA3 Institute examination.

No reviewed source shows that test-to-deployment evidence changes insurance terms

Underwriting AI-related risk means insuring losses that arise from an organisation’s use of AI. It differs from AI in underwriting, where an insurer uses AI to make its own decisions; this article concerns the first. An insurer considering such losses may ask how a system operates and which controls are supported by records, and the organisation can respond with test configurations, observed failures and explicit differences from deployment. Whether insurers request or value those records is not established by the materials reviewed. ODA3 Institute’s proposition is that bounded evidence could make exposure descriptions more specific.

There is relevant but separate US market context. An Insurance Journal report of 17 August 2026 describes three Insurance Services Office (ISO) endorsements (amendments attached to a policy) that exclude, meaning remove cover for, losses arising out of generative AI under US general liability coverage, using descriptions provided by ISO’s parent, Verisk [3]. An attorney quoted in the report saw insurers increasingly filing to use them and expected rapid adoption of some form of AI exclusion in most lines, while saying that actual adoption and wording are not yet known and that some insurers may instead underwrite and price the exposure [3]. These are attributed expectations, not demonstrated market outcomes.

Where an applicable exclusion removes the exposure, better AI test evidence may not change the coverage position. Where an insurer chooses to underwrite the exposure, bounded records could inform its inquiry, but no such use or pricing benefit is verified. None of this shows that the endorsements appear in an energy operator’s policy, that they govern property or business-interruption cover, or that a particular AI-related event would be excluded. The endorsement forms and any issued policy were not examined, and non-US wording was not reviewed.

The responsibilities are distinct. Management provides accurate descriptions and records. A suitably commissioned, independent examiner assesses bounded evidence. Insurers decide underwriting and pricing. Any coverage dispute turns on the wording, the facts, applicable law and, where relevant, adjudication. Technical failure, financial loss, legal liability and policy response are separate questions.

What executives should decide

The decision is whether to approve, restrict or seek further examination before relying on a test result for a changed deployment. It does not require a universal evaluation format.

Action Responsible party Required decision or record Revisit when
Define what the AI system is authorised to do COO or accountable operating executive, with the AI system owner Advisory, assistive or action-capable role; named approver Tools or permissions change
Keep evaluation evidence tied to the deployed version, including records requested from any supplier that ran the test CISO / AI governance lead Versioned test results, observed failures and material configuration differences The application, environment or evidence changes
Define acceptance criteria before relying on results AI claim owner and relevant independent reviewer The exact claim, scope, exclusions and unresolved assumptions New evidence changes the supportable conclusion
Document operator authorisation separately from AI evidence examination COO / OT engineering authority Plant-release record under the operator’s engineering process; list of unresolved AI-specific reliance limits AI role, operator workflow or safety-relevant interface changes
Check the actual insurance position before making exposure claims Head of Risk Transfer / General Counsel Policy and endorsement wording; broker or insurer confirmation of which AI-related exposures are underwritten, excluded or unresolved Renewal, endorsement change or material change in AI authority

Notably Absent

The public DOE pages checked do not provide a Stormbreaker test report for an identified utility, a plant-deployment comparison, a simulator-fidelity assessment, an independent assurance finding for an operational installation, or a specification of the records the platform produces [1][2]. They do not state that configuration transfer is a plant safety approval criterion.

The insurance report does not establish underwriter reliance on a testbed comparison, premium reductions attributable to such evidence, or coverage treatment for an AI-assisted energy incident [3]. It reports insurer interest in exclusions and an alternative of pricing the exposure, not verified universal adoption. No court ruling on how these exclusions apply to an AI-assisted energy loss, and no AI-attributed energy-sector claims data, were located in the sources checked; the attorney in [3] says no case yet shows how courts will read the exclusions. These are limits of the materials reviewed, not claims that such evidence or decisions exist nowhere.

Conclusion

DOE states that AI evaluation outcomes depend on the configured system and the conditions of testing [1]. ODA3 Institute’s analysis is that organisations should keep the evidence behind an AI-related claim traceable when tools, permissions and deployment conditions change. The operator remains responsible for plant safety and operational release; independent AI examination can address only the claims its evidence is sufficient to support. Whether insurers will use such records remains an open question.

References

[1] US Department of Energy, Office of Cybersecurity, Energy Security, and Emergency Response. “CESER Releases New Testbed to Advance LLM and Agentic AI Evaluation for Critical Infrastructure.” Energy.gov, 16 July 2026. https://www.energy.gov/ceser/articles/ceser-releases-new-testbed-advance-llm-and-agentic-ai-evaluation-critical (accessed 23 October 2026).

[2] US Department of Energy, Office of Cybersecurity, Energy Security, and Emergency Response. “CESER and Lawrence Livermore National Laboratory Launch AI Testbed to Strengthen the Energy Sector’s AI Cybersecurity.” Energy.gov, 14 April 2026. https://www.energy.gov/ceser/articles/ceser-and-lawrence-livermore-national-laboratory-launch-ai-testbed-strengthen-energy (accessed 23 October 2026).

[3] Jergler, Don. “Insurer Interest in AI Coverage Exclusions Growing as Risk Becomes Omnipresent.” Insurance Journal, 17 August 2026. https://www.insurancejournal.com/magazines/mag-features/2026/08/17/881424.htm (accessed 23 October 2026).

Methodology note

Source-bounded examination of test configuration, observed behaviour, deployment differences, and reliance limits; not a completed independent examination