Something unusual is happening at the frontier of AI. OpenAI has slowed parts of its frontier model development while strengthening monitoring, alignment, and security. Dario Amodei is calling for “pacing the frontier.” Sam Altman agrees, and Demis Hassabis has broadly supported the direction.

The specifics differ, but they point to a common concern: as AI becomes more capable, our ability to understand and assure its behavior has to keep pace.

There is a paradox at the heart of this. We make AI more capable precisely so that it can handle situations we didn’t anticipate in advance—to interpret unfamiliar circumstances, reason through ambiguity, adapt its approach, and determine an appropriate response.

But the better AI becomes at determining behavior for situations we didn’t anticipate, the harder it becomes to completely anticipate and test that behavior before deployment.

That is the AI Assurance Paradox. It applies directly to the AI applications enterprises are putting into production, from customer-service chatbots to autonomous agents, because a fundamental change is taking place in software: part of application design is moving into runtime.

Design used to happen mostly before runtime

For most of software history, there has been a useful separation between design time and runtime. Engineers decide what software should do, encode those decisions into logic, test the resulting system, deploy it, and monitor whether it executes correctly.

Complex software can certainly behave unexpectedly as components fail, configurations change, and interactions produce unforeseen system effects. Yet its underlying behavioral logic is largely specified before deployment.

Generative AI changes that relationship. Consider something as simple as a customer-support chatbot. Developers might specify its instructions, knowledge sources, company policies, escalation rules, and constraints, but they don’t completely specify how the application should behave in every possible customer interaction.

At runtime, the model determines how to interpret a particular request, which information matters, how a policy applies to that situation, and what response to construct. For a novel customer situation, the precise behavioral path may never have existed before that interaction.

The model might retrieve the correct policy but misapply it, unnecessarily deny a valid request, keep answering when it should escalate, or combine individually correct facts into misleading guidance. The underlying infrastructure can remain healthy throughout the interaction: the APIs work, the model responds, and the security controls remain intact. The failure is in behavior that was assembled at runtime.

The business consequences can be significant. A customer can be incorrectly denied service, a support interaction can fail to resolve, an incorrect business decision can be made, or an AI application can technically complete a task while producing the wrong business outcome.

Agents push more design into runtime

Agents extend the same shift from generating responses to determining strategies and actions. Instead of specifying every step, we increasingly give an AI system an objective, tools, permissions, context, and constraints and allow it to determine how to accomplish the objective.

The agent can decide which steps to take, which information matters, which tools to invoke, when to retry, when to change approach, and when to escalate. These decisions increasingly happen during execution rather than being completely encoded beforehand.

There is therefore a continuum. A generative AI application constructs portions of its response behavior at runtime. An agent can additionally construct portions of its strategy and action sequence at runtime. As models become more capable, the range of situations in which we rely on them to make these determinations grows.

The property that makes AI useful also creates the assurance paradox

Increasing model capability is valuable precisely because we don’t want to specify every possible situation in advance. We want AI to handle novelty—to reason across unfamiliar circumstances, combine information, adapt its approach, recover from problems, and find ways of accomplishing objectives that developers didn’t explicitly program.

Greater capability can produce dramatically better outcomes. It can also expand the range of situations in which the AI itself determines what appropriate behavior should be.

That brings us back to the paradox:

The better AI becomes at determining behavior for situations we didn’t anticipate, the harder it becomes to completely anticipate and test that behavior before deployment.

Pre-deployment evaluation therefore faces an inherent coverage boundary. We can test behaviors we know to look for under conditions we know how to construct. Yet an important reason to deploy AI in the first place is its ability to handle situations and combinations we didn’t completely anticipate.

As more consequential behavior is determined during operation, assurance has to extend into runtime as well.

The OpenAI–Hugging Face incident makes this concrete

The recent OpenAI–Hugging Face incident provides an unusually clear example because it occurred during an evaluation. OpenAI reported agents circumventing intended controls, establishing unauthorized communication channels, exploiting shared infrastructure, gaining internet access, and sharing discoveries across otherwise separate environments.

The agents discovered consequential behavioral paths through their environment during execution that the evaluation designers had not intended. OpenAI’s response included stronger workload and network isolation, continuous security testing, and expanded monitoring of sequences and tool actions.

That makes the incident particularly instructive for the assurance paradox. An environment built to evaluate the AI itself became a system whose runtime behavior also needed to be assured. As AI gets better at figuring out how to accomplish an objective, understanding the behavioral paths it actually takes becomes increasingly important.

The same principle applies well before systems reach this level of autonomy. A chatbot determining how to apply a policy to an unfamiliar customer situation and an agent discovering a novel sequence of tools represent different points on the same continuum: consequential application behavior is increasingly being determined during execution.

Frontier labs can assure their models. They can’t assure your system.

Frontier labs are responding to this challenge with stronger evaluations, monitoring, alignment, security, and other safeguards. Those efforts are essential, and the labs have visibility into their models and development environments that enterprises will never have.

Their assurance boundary, however, ends before the complete enterprise application. An enterprise combines a model with its own instructions, proprietary data, retrieved context, business policies, tools, permissions, workflows, and operating environment. Those elements materially shape the behavior that emerges in production.

Even a thoroughly evaluated frontier model therefore becomes part of a new behavioral system when an enterprise deploys it. The model provider cannot pre-evaluate every customer situation, policy interpretation, retrieved context, tool interaction, and business condition that system will encounter.

Frontier labs must assure the models they build. Enterprises must assure the systems they operate.

Pre-deployment evaluations remain an essential part of that enterprise responsibility. Their role becomes clearer when we recognize the assurance paradox: they test anticipated behaviors and scenarios before deployment, while runtime assurance covers behavior that emerges as AI encounters the real operating environment.

What pacing the frontier means for enterprises

Frontier labs are debating how quickly they can safely advance model capability. Enterprises don’t control that pace, but they do control how those capabilities are incorporated into applications, workflows, and business decisions.

The lesson for enterprises is to evolve the architecture of assurance along with the architecture of AI. When developers specified most application behavior before deployment, much of assurance could reasonably concentrate there. Generative AI changes that boundary because meaningful parts of application behavior are now determined during execution.

This begins with chatbots constructing situation-specific responses and becomes more consequential as agents construct strategies and actions. The flexibility is precisely where much of AI’s value comes from, and it also changes where assurance has to happen.

As design moves into runtime, assurance has to move with it.

Frontier-model evaluations remain essential. They cannot provide complete assurance for behavior that will only be determined when AI encounters the enterprise’s real-world environment.

Frontier labs must assure the models they build. Enterprises must assure the systems they operate.