When AI Controls a Drone: What Physical AI Security Requires Beyond Model Safety

Methodology Note

This publication analyzes publicly available primary disclosures from Anthropic and applies ODA3 Institute's analytical methodology to distinguish verified facts, analytical assessments, and remaining uncertainties. Framework mappings are evidence-bounded analytical interpretations of publicly available information and should not be interpreted as evidence of certification, compliance, regulatory approval, or product evaluation.

Executive Takeaways

• Strong component-level performance does not guarantee safe end-to-end physical autonomy.
• Reconstruction errors can propagate into real-world kinetic actions.
• Average-case reliability is a more meaningful operational indicator than best-case demonstrations.
• Independent controls between model output and actuation remain essential.
• PAI-SF™ analyzes this assurance chain through the Kinetic Zero Trust principle.

On July 24, 2026, Anthropic and Andon Labs published results from Project Pilot, a benchmark called Drone-Bench that evaluates whether AI models can autonomously fly a quadrotor drone to locate and follow a person indoors. The best-performing model, Claude Fable 5, exceeded a human-AI team baseline on every constituent sub-task except one — reconstructing a 3D map of the environment from video. That single weak link was enough to produce a documented failure on the physical drone itself: the model, acting confidently on its own flawed reconstruction, flew into a wall it had mapped as a doorway.

Anthropic reported no harm in its published account, and is explicit that the test environment was slow-speed, indoor, and limited to a single floorplan with a small, consenting group of people. But the finding is a clean, publicly documented illustration of a problem that physical AI security has to solve directly: a model can demonstrate strong performance across most components of a physical task and still fail, with physical consequences, once those components are chained together into real-world action. This is precisely the assurance question PAI-SF™: Physical AI Security by Design is built to address, anchored on a single controlling principle: Kinetic Zero Trust. Under Kinetic Zero Trust, no model-generated command receives physical authority merely because it originated from an authenticated model, agent, or control channel. Physical action must remain independently constrained, authorized, observable, and interruptible — regardless of how confident, capable, or well-tested the model issuing the command is.

What happened

Working again with Andon Labs — the same partner behind Anthropic's earlier Project Vend and Project Fetch work — Anthropic's Frontier Red Team developed a new evaluation assessing AI models' ability to use a flying drone to perform a locate-and-follow task of the kind used in aerial surveillance. Andon Labs, which built and independently ran the benchmark, decomposed the objective into five sub-tasks: Reconstruct (convert office video into a 3D model and a 2D obstacle map), Localize (match the drone's current view to a position on that map), Navigate (plan and fly a path between rooms, continuously re-localizing to correct for drift), Detect (find a designated person from a reference photo), and Follow (keep that person centered in view at a stable distance).

Fifteen models from three developers were tested — including several generations of GPT, Gemini, and Claude models, up to GPT-5.6 Sol and Claude Fable 5. Performance was measured against a baseline built by human-AI teams using off-the-shelf algorithms for each sub-task, not against an idealized ceiling. The clear trend across model generations was steady improvement on detection and following, with reconstruction and localization lagging well behind.

Claude Fable 5 was the strongest performer, clearing baseline on every sub-task except reconstruction. On the physical drone — a $129 DJI Tello EDU — it outperformed the baseline noticeably at detecting and following the target. But reconstruction errors compounded through localization and navigation, and the model was unable to autonomously navigate between rooms. Anthropic's own published footage shows the failure directly: Fable 5 confidently flew the drone into what its own reconstruction told it was a doorway, but was in fact a wall.

Anthropic separately reports a consistency gap worth treating as its own finding, independent of the wall incident: across ten simulation runs, models overall reached human-baseline performance in at least one run for four of the five sub-tasks, but Claude Fable 5 — the strongest performer — reached that baseline on average for only three of the five, and that average-case consistency lagged roughly six months behind the point at which best-case performance first cleared the same bar. In other words, the frontier of what a model can do is measurably ahead of what it reliably does do.

What the evidence establishes

Confirmed, from Anthropic's own published research: the benchmark design and its five sub-tasks; the 15-model, three-developer test set; Claude Fable 5's status as the strongest performer, exceeding baseline on four of five sub-tasks; models overall reaching baseline in at least one run for four of five sub-tasks, with Fable 5 averaging baseline on three of five; the specific reconstruction-driven navigation failure, including the wall/doorway incident, shown in Anthropic's own video footage; the roughly six-month gap between best-run and average-run consistency; and the stated test limitations — slow flight speeds, a single indoor floorplan, a small number of people, no outdoor or large-crowd testing.

Anthropic's own analytical framing, presented as the company's view rather than an independent finding: the explicit parallel to agentic coding, where human approval of "nearly every tool call" gave way within months to much greater autonomy, and the company's own statement that "once models pass capability and reliability thresholds... there will be real pressure to treat human oversight as a cost rather than a safeguard."

Not established by this research: performance in outdoor, high-speed, crowded, or adversarial conditions; resistance to deliberate sensor manipulation or spoofing (the benchmark measures capability and reliability under benign conditions, not adversarial robustness); reliability sufficient for any operational deployment, safety-critical or otherwise; and whether the reconstruction bottleneck will close in the next model generation, though Anthropic characterizes it as "the missing piece" given capability already demonstrated elsewhere in the task chain.

Why this matters for operational AI security

The Drone-Bench finding is best read as evidence about a pattern, not a single incident: a model can demonstrate strong performance across most components of a physical task and still fail when those components are chained into continuous real-world action, because errors from an early step (reconstruction) propagate and compound through later, otherwise-capable steps (localization, navigation). This is not unique to drones. Any physical AI system that perceives an environment, builds an internal representation of it, and then acts on that representation carries the same structural risk: a plausible-looking, confidently held world-model can be wrong, and the system may have no independent way to notice.

An organization should treat this as directly relevant to the extent it operates, or is evaluating, AI systems that translate model-generated perception or planning into physical actuation — drones, mobile robots, autonomous vehicles, or any system where the independence and effectiveness of the controls between "the model decided" and "the hardware moved" must be established. It is less directly relevant to purely advisory or human-approved physical AI use, where a person reviews the model's output before any physical action occurs.

What security and governance leaders should examine

Immediate review. Identify every physical AI system in your environment where model output converts into unmediated physical action, and confirm whether that conversion passes through any independent validation layer or is trusted directly.

Near-term validation. Test sub-tasks and full task chains separately, not only in isolation — Drone-Bench's central finding is that per-task competence does not predict chained, end-to-end reliability. Establish release or deployment criteria based on average-case reliability across repeated runs, not best-case demonstration; Anthropic's own six-month gap between the two is a useful benchmark for how large that difference can be. Test human-override latency and reliability under realistic operating conditions, not only under calm, cooperative test conditions.

Longer-term governance and assurance. Build an explicit, tested safe-state behavior for every physical AI system — a defined, deterministic response (stop, hover, return, land) that triggers when perception or localization confidence drops below a defined threshold, rather than allowing the system to continue acting on a low-confidence world-model. Treat this as a control to be demonstrated under adversarial and degraded conditions, not merely documented in a design specification.

PAI-SF™ and Kinetic Zero Trust

Drone-Bench is a useful case study precisely because the failure is not exotic. Fable 5 was not attacked, jailbroken, or manipulated — it simply held a confident, incorrect belief about its physical surroundings and acted on it. That is exactly the scenario Kinetic Zero Trust is designed for: it assumes that any model, however capable, will sometimes be wrong about the physical world, and it requires that physical authority never rest on the model's own confidence in its command.

What distinguishes PAI-SF™ from a general cybersecurity framework applied to robotics is that it treats the full path from cognition to consequence as one accountable chain, not a series of separately governed systems: perception → model decision → autonomy boundary → kinetic command → physical effect → monitoring → intervention → recovery. In Drone-Bench, the reconstruction error propagated through localization and navigation into physical action; the public report does not disclose what independent validation, if any, existed between the model-generated navigation output and actuation. The chain framing is what makes that joint visible as a distinct, assessable point to examine, rather than a diffuse "AI safety" concern.

Read against PAI-SF™'s domain structure, several are directly engaged by this specific finding:

Sensor and Perception Integrity. The reconstruction failure is, at root, an internal world-model or state-estimation failure — the model's 3D map of the environment did not match reality. Mapped analytically onto PAI-SF™, this is a Sensor and Perception Integrity concern: the public report does not describe whether the system had any independent mechanism to flag that its own reconstruction might be unreliable before acting on it.

Actuator and Kinetic Command Safety. The public report does not describe whether a separate control layer independently validated the model-generated navigation command — flying toward what the model believed was a doorway — against the drone's other available sensor data before actuation. This is the specific gap Kinetic Zero Trust targets: a command should not be trusted merely because it originates from the model's own world-model, but should be independently verified against the physical environment before it is executed.

Autonomy Boundaries. Anthropic's own commentary — about pressure to treat human oversight as a cost once capability thresholds are passed — is precisely a question of where autonomy boundaries should sit and how deliberately they should be revisited as capability increases, rather than eroding by default as reliability appears to improve.

Safe-State and Degraded-Mode Behavior. What is documented is that the reconstruction error propagated into navigation and produced the reported wall/doorway failure. The public account does not disclose whether confidence-based pausing, hovering, or other degraded-mode controls were configured or activated. A defined degraded-mode behavior — hovering, halting, or requesting reconstruction retry before continuing — is the direct control this finding argues for testing, regardless of what the tested system actually did or did not have in place.

Human Override and Intervention. The test was conducted under close research supervision, but a deployed system would need a proven, low-latency override path; Drone-Bench does not test this dimension at all, and that is itself a gap worth naming rather than assuming closed.

Runtime Monitoring and Telemetry. Decomposing the benchmark into sub-tasks is itself a form of monitoring discipline — it lets researchers see exactly where the chain broke rather than only observing an undifferentiated end-to-end failure. Runtime telemetry that surfaces sub-task confidence in a live deployment is directly analogous.

Simulation, Testing and Validation. Andon Labs' approach — reproducing the physical task in software to allow many rapid iterations before physical testing — is a sound instance of this domain in practice, though Anthropic itself notes that simulation performance does not fully predict real-world reliability, since the physical demonstration surfaced the wall/doorway failure that simulation-only testing might not have caught in the same way.

Functional Safety Interface. The public report does not describe whether an interface existed between the AI model's navigation output and a separate, deterministic safety layer. Whatever the tested architecture actually contained, the observed outcome — a confident misperception converting directly into a physical collision attempt — is the failure mode a functional safety interface exists to prevent.

Physical Operating Environment. Anthropic's own stated limitations — slow speeds, one floorplan, no outdoor or crowd testing — are a direct acknowledgment that operating-environment complexity was deliberately constrained, and that reliability claims do not yet extend beyond it.

GAISSF™, UAIF™, and AI-IRF™ context

PAI-SF™ is the primary lens for this development, but it sits within the broader GAISSF™ Ecosystem. GAISSF™ frames the governance question underneath Anthropic's own commentary: who decides when a physical AI system has passed a capability and reliability threshold sufficient to reduce human oversight, and what evidence that decision should require. UAIF™ offers the vocabulary to classify a physical AI failure like this one without overstating it — separating a benign, disclosed research finding (this case) from a genuine incident with real-world impact (not what occurred here) — mechanism (reconstruction error), affected layer (perception and navigation), confirmed impact (a failed navigation attempt in a controlled test), and explicitly no confirmed harm. AI-IRF™ becomes directly relevant only once physical AI systems are deployed operationally; its safe-state, containment, and recovery-validation lifecycle stages map onto exactly the safe-state and recovery-validation gaps this benchmark surfaces, though this was a research finding, not an incident requiring response.

Framework mapping is analytical and evidence-bounded. It does not establish compliance, certification, regulatory approval, or that use of any GAISSF™, UAIF™, AI-IRF™, or PAI-SF™ framework would have prevented this specific research outcome.

Notably Absent

At the time of publication, the available public evidence does not establish: performance under outdoor, high-speed, or crowded conditions; resistance to deliberate sensor spoofing or adversarial manipulation, as distinct from benign reconstruction error; whether the reconstruction bottleneck will be resolved in the next model generation; any tested human-override latency or reliability figure; or any standards body's finalized technical requirements for physical AI perception-to-actuation assurance. Absence of public evidence on these points should not be read as evidence that solutions do not exist — only that they have not yet been demonstrated in this published research.

What to watch next

Watch for: further Anthropic or Andon Labs publications addressing the reconstruction bottleneck specifically; any follow-up testing in outdoor, adversarial, or higher-speed conditions; independent replication of Drone-Bench by other labs or research groups; and standards-body work specifically addressing perception-to-actuation assurance for physical AI systems, which ODA3 will report on once a specific, verifiable work item can be confirmed against an authoritative source.

Conclusion

Fable 5 did not fail because it was unaligned or adversarially manipulated. It failed because it held a confident, wrong belief about its physical surroundings, and the public report does not describe any mechanism in the tested system that stopped that belief from converting into a physical action. That is the operational lesson: model capability, however impressive, is not the same claim as physical safety, and the gap between the two is exactly where Kinetic Zero Trust is meant to sit — physical authority independently constrained, authorized, observable, and interruptible, never granted on the strength of the model's own confidence alone. Meaningful uncertainty remains about how this finding generalizes beyond a single indoor floorplan and slow-speed flight, and that uncertainty should temper any claim about present-day operational readiness in either direction.

As physical AI systems move from research demonstrations like this one toward operational deployment, the organizations fielding them will need more than a general awareness that capability is not the same as reliability — they will need testable controls, implementation evidence, and independent assurance at each joint in the chain from perception to physical consequence. That is the gap PAI-SF™ is built to close.

This analysis is provided for research and operational-security purposes. It does not constitute legal advice, establish compliance, or represent regulatory approval.

Review the PAI-SF™ domain structure and assessment methodology to evaluate your organization's physical AI security posture.

Related ODA3 resources: PAI-SF™ overview · GAISSF™ overview · UAIF™ overview · AI-IRF™ overview · GAISSF™ Ecosystem page

Sources and evidence note

Evidence & Analytical Status Ledger

Fact-check and claim-verification ledger

• Test date, benchmark structure, 15 models/three developers, Fable 5 as best performer, wall/doorway footage, six-month consistency gap, and stated test limitations: all confirmed directly against Anthropic's published research page — T1.
• The DJI Tello EDU / $129 detail: confirmed via Anthropic's own footnote.
• The four-of-five best-run figure is attributed to models generally, per Anthropic's own text; only the three-of-five average-run figure is attributed specifically to Fable 5.
• No standards-body work item is cited in this article for physical AI perception-to-actuation security; ODA3 will report on this once a specific, independently verifiable work item can be confirmed against an authoritative source.
• The PAI-SF™ domain mapping in this article uses ODA3's canonical 12-domain architecture and its Kinetic Zero Trust principle.

© 2026 ODA3 Institute. All rights reserved.