One Update, 8.5 Million Machines: The WEF Says AI Safety Lives «After the Lab». For a Small Team, That Is an Engineering Spec

On 19 July 2024, at 04:09 UTC, a security vendor pushed one configuration file to Windows hosts, and 8.5 million Windows machines went down. No model was involved. It is still the clearest picture I know of what the World Economic Forum means by AI safety «after the lab».

On 3 November 2026 the Forum published Why AI safety depends on what happens after the lab, by Audrey Duet of the WEF and Philippe Cordier and Ines Bedar of Capgemini. Its argument: the harm a failure does depends on what the system can do, the authority it holds, how widely it runs and how fast someone can contain it. It alludes to the CrowdStrike outage without naming it, and says it was no AI failure.

I read it from the deployer's chair. A good portion of the work we do is wiring agents into somebody's CRM, ticket queue or billing API. For the 2 to 15 person company putting its first agent near customer data, the «deployer» the WEF addresses is you, and there is no compliance department behind you. From that seat, the WEF's policy reads as an engineering specification.

Safety is a property of the system you deploy, not of the model you bought

A model can pass every lab test and still behave differently once connected to real tools, data and infrastructure, because sandboxes rarely reproduce a deployment. So the WEF makes the deployed system the unit of safety, and blast radius its measure.

Microsoft's 20 July 2024 post put the outage at 8.5 million Windows devices, less than one percent of all Windows machines, and tied the economic and societal impact to the enterprises running critical services on CrowdStrike. Blast radius has four variables, and the device count is only one:

1. Capability: what the component can do. The sensor ran with enough privilege to crash the boot.
2. Authority: whether anyone had to approve this action. The update went out without a canary; CrowdStrike's post-incident review then committed to staggered rollouts that start with one.
3. Reach: how many systems take the change at once. Every Windows host running the sensor that was online in that window.
4. Time to contain and recover: CrowdStrike remediated the file at 05:27 UTC, 78 minutes after release. Machines that had already loaded it needed manual remediation, one by one.

Each variable is set in the code around the model. I keep arguing that the model is the commodity and the harness around it is the moat. The WEF now locates safety in the same place. It is the vendor-dependency lesson of the outage with an agent in the vendor's slot.

The WEF piece contains no numbers, and two of its three authors work for an integrator

The article has no statistic and no incident count; incidents «multiply», by how much it does not say. For a piece that argues for measurement, that is a weakness.

The one global counter it names shows why. The OECD AI Incidents Monitor is built from news coverage, and on 8 November 2026 it listed «about 18873» incidents and hazards. A monitor that reads the press sees the failures that reached the press, and an agent that wrote the wrong row to a ledger rarely does.

Then there is who is talking. Capgemini sells integration, and the instrument the article cites, the Agent Capability and Authorization Profile, comes from the WEF's May 2026 playbook AI Agents in Action, written «in collaboration with Capgemini». Conectia sells integration too, and I earn money building the harness this post recommends. Weigh both of us accordingly; the argument can be checked against your own systems.

A permission manifest per agent is the small-team version of the WEF's authorization profile

The playbook defines the profile as «the organization's authorization for an AI agent to act in a specific workflow, defining permissions, checkpoints, owners and evidence». Two of its rules are worth taking as they are: list every MCP connection with its scope and revocation conditions, and let a downstream agent run only under the intersection of its own permissions and its caller's. It also concedes the profile is not yet machine-readable and «should evolve» towards policy-as-code.

For an eight-person team, that is a versioned file in the repository, one per agent per workflow, that a gateway reads at runtime:

• Tools and scopes: each tool, with the narrowest scope that works. Read on contacts, write only on draft notes.
• Principal: whose authority the call carries. A user's delegated token and a shared service account give different answers when someone asks whose request it was.
• Limits: rate, spend and records touched per action.
• Checkpoints: the actions that wait for a named human.
• Owner and revocation: one accountable person, and the exact command that revokes access.

If every tool call goes through a gateway that reads the manifest, the manifest describes the system. In a wiki, it describes your intentions. That is the least-privilege-per-tool rule for securing tool-using agents, with a governance name.

An audit trail earns its cost the day it tells a model failure from an integration error

The WEF asks for audit trails of what the system did, which tools it used, when and on whose authority, because without them nobody can tell a model failure from an integration error, a weak safeguard or a bad deployment decision. In the enterprise postmortems that went worst for me, the log lines existed. What was missing was the state of the world at that minute: which config version was live, which credential the call used, which deploy had landed ten minutes earlier.

So the record for each agent action carries: a timestamp from one clock source; a trace ID across every hop; the model and prompt versions; the manifest version in force; the principal; the tool, its arguments (redacted where they hold personal data) and the response; and the decision, allowed, denied or sent to a human, with the approver's name. With that, «did the model invent a refund or did our API accept a malformed one» becomes a query you can run.

The denied tool call is your near miss. The WEF points to aviation, where confidential, no-blame reporting of near misses helped make flying safe; SRE teams run the same idea as the blameless postmortem. A gateway refusing an out-of-scope call is the cheapest safety data you can collect. Count the refusals and read them weekly, with nobody's name on the agenda.

For a deadline, borrow the EU's. Article 73 of the AI Act gives providers of high-risk systems 15 days from awareness to report a serious incident, two days for a widespread infringement or a serious and irreversible disruption of critical infrastructure, ten days where a person has died; Article 26 has their deployers keep the logs under their control for at least six months. Most small-team agents are out of scope, and the test still works: could you write a factual account of your agent's bad afternoon within two days?

A kill switch has to stop the actions already in flight, and recovery has to exist before the incident

The WEF says containing and recovering matters as much as preventing. The outage shows it: the file was pulled after 78 minutes, and recovery ran far longer.

For an agent, the kill switch is one control that revokes its credentials at the gateway, drains its queue and routes pending work to a human. Suspension triggers are written before launch: denied calls above a threshold, spend over budget, an action rate far above baseline, and any write outside the manifest's scope, which should be impossible and so stops the agent automatically. Recovery is designed with the write path: idempotency keys so a retry cannot pay twice, a compensating action for every write, soft deletes where the data allows. The checkpoint before anything irreversible is the same control inside a single loop.

Scale the safety case to the blast radius, or «require evidence» becomes a paperwork tax

The strongest objection to the WEF's third priority is fair: a safety case in the aviation or nuclear sense is disproportionate for most small deployments, and «require evidence» without a shared standard becomes whatever the next auditor asks for. Bruegel has priced EU AI Act compliance at 9 to 17 percent of a €170,000 build, as I covered in regulating AI after it ships. A ten-person company is not paying that per agent.

The WEF answers its own objection: scrutiny should follow impact radius, reversibility, autonomy and access, from self-assessment for a document summarizer to independent testing for an agent that moves money. Here is how I tier it:

At ten people, «independent» means a contractor or a peer company trading reviews. You would build most of these anyway the first time a customer asks why your agent did something.

Five things I would ship this quarter before an agent gets write access

1. Write a manifest for every agent in every workflow, with tools, scopes, principal, limits, checkpoints and a named owner.
2. Route every tool call through one gateway that enforces the manifest and writes the record. One choke point beats five integrations that each log differently.
3. Review denied calls weekly as near misses, blameless, thirty minutes.
4. Drill the kill switch monthly and time it, from trigger to the last action stopped.
5. Tier every agent by blast radius and attach the evidence its tier needs, then re-run it whenever the model, the prompt or the permissions change.

One update reached 8.5 million machines because its capability, authority and reach were all set to the maximum, and the way back needed hands on each machine. The model is the commodity and the harness is the moat; since 3 November 2026 the WEF has put safety in the harness too, so whoever builds the harness owns it. If you are about to give an agent write access and want a second pair of eyes on its manifest, talk to us.