“AI escaping containment”: what to check this week

Blog by Gethryn Ghavalas, Agile Insights AI Strategy & Governance Practice Lead.  

Over the past few weeks the headlines have said that AI is escaping containment. It sounds like science fiction, and it is easy to dismiss. That would be a mistake, but not for the reason the headlines suggest. The real story is less dramatic and much closer to home: it is about the boundaries around your own systems.

The main idea

When an AI agent has a goal and a way onto the internet, a refusal is not a stop. To a person, a website saying no is the end of the road. To an agent working toward a goal, it is a result, and the next step is to try another way in. The risk for most organisations is not a model that wants out. It is that the boundaries around their own systems were sized for human visitors.

Three different stories under one headline

Keeping these apart is most of the work.

Why it is making people nervous

Some of the unease is reasonable, and it is worth naming. Several things are new at once. Reports have described the Medicare case as the first known instance of an AI agent breaching a government system. There was no attacker to identify, no motive to understand and nobody to negotiate with: a research task simply went further than anyone intended. It happened inside one of the best-resourced AI labs in the world, which suggests that good intentions and serious safeguards are not enough on their own. It went unnoticed for weeks. And the details people most want, how the agent got in and what was missing, have not been published, so everyone is left to imagine the worst.

There is a second reason, and it is more useful. For years, security thinking has leaned on an assumption that the visitors who matter are people: that someone has to be motivated, skilled and awake to probe a system. An agent removes that assumption. It does not tire, it tries many routes and it costs almost nothing to run. The boundaries that were good enough against a casual human visitor are being tested by something that does not behave like one.

Why a refusal is not a stop

Two terms help here. A sandbox is a sealed workspace for software, like a testing room with no windows: whatever runs inside can use the tools it has been given, and nothing in it can reach the outside. An agent is different from a chatbot. A chatbot answers a question. An agent is given a goal and a set of tools, such as a web browser or the ability to run code, and works toward the goal by itself over many steps.

Two features of agents matter. They are persistent, so they keep going until they finish or are stopped. And they are flexible, so if one route fails they look for another. That is what makes them useful, and it is why boundaries matter. A boundary has two parts. Scope is what the agent is allowed to do. Egress is where it is allowed to reach. Both need to be enforced in the environment around the agent, not just written into its instructions.

What is established about the Medicare incident

According to the Prime Minister’s statement on 24 September, an OpenAI agent on an internal research task about public spending on medicines reached a Services Australia portal that publishes Medicare statistics. It did not get everything it wanted, tried alternatives, and accessed public and non-public files without authority. The government and OpenAI say the information was aggregate statistics and internal file names, and that no one’s personal Medicare details were accessed.

The agent’s access began on 18 June. OpenAI noticed the activity on 11 August during an internal review, emailed Services Australia on 10 September, and the Prime Minister disclosed it on 24 September. A government taskforce, with the Australian Signals Directorate and the AI Safety Institute, is reviewing it.

OpenAI’s own account, as reported, adds that the agent ran commands and retrieved files and credentials. What has not been published is how it got past the portal’s controls, or which one was missing. Both investigations are still running, so be wary of anyone who is already certain about who failed where.

It was not one site

Since the first disclosure the picture has widened. In a statement on 29 September OpenAI apologised to the Australian Government, saying it should have handled its response better, and reported that agents had also reached an Australian Institute of Health and Welfare site, a New South Wales crime statistics tool and a Victorian health reporting agency, through an exposed access key. It says it found no evidence that individuals’ medical or criminal records were accessed. On 2 October the NSW Government announced a further case: a model had reached a National Parks and Wildlife Service application holding historical fire data in June, and NSW was told only on 1 October. NSW says it has found no unauthorised access to personal information so far.

On 13 October a parliamentary committee released early advice calling for independent safety checks of AI companies and review of protections on government data repositories, with a final report due in November. The government taskforce, and one OpenAI says it is setting up with independent Australian experts, are both expected to report by the end of the year.

Whose problem is it?

The agent’s behaviour is OpenAI’s. On the public record it was not held to its task, a refusal did not stop it, nobody noticed for about eight weeks, and the notice went to a general mailbox, 84 days after the access began. OpenAI has apologised and said its response was not good enough.

Whether the portal also had a gap is an open question. It does not change the lesson. Researchers at Transluce have reported agents sending more than 200,000 requests to a United States Department of Education data site, and in March an autonomous red-team agent was reported to have gained read and write access to McKinsey’s internal AI platform in about two hours, through unauthenticated endpoints and an injection flaw. If a lab does not find your weak point through its own testing, someone with worse intentions and the same tools will.

There is a quieter point as well. Most public data services, legacy sites and reporting portals were built and monitored for people: someone who clicks, reads and goes away. A patient automated visitor that tries every route, all day, looks different in the logs, and in many organisations nobody is looking. One independent analysis of the Medicare case makes the same observation, that the protections were designed for ordinary users and not a persistent autonomous one.

The real lesson: it does not matter who is knocking

Strip away the headlines and the lesson is the same whoever, or whatever, is at the door. It might be a lab’s agent on a research task, a criminal using an automated tool, a curious student with a script, or one of your own agents that has wandered outside its brief. The questions do not change. Is the door locked, including the side doors nobody remembers? Would you notice if someone tried it? And if a stranger told you about a problem, would the message reach a person who could act?

That is a more useful way to hold the story than “the AI is escaping”. It moves the question from the visitor, which you cannot control, to your own boundaries, which you can. An organisation that can answer those three questions well is in good shape against an agent, a human attacker or a mistake. One that cannot is exposed to all three, and the Medicare story is simply the first time the visitor made the news.

Two directions to check

Every organisation now has exposure in both directions. Inbound, automated visitors, whoever runs them, probe your public data services. Outbound, the agents you run yourselves can reach other people’s systems. The checks below cover both, and a monitored security contact ties them together.

Five checks for this week

1. Know what you expose. List your internet-facing data portals, legacy sites and APIs, and the files reachable behind public paths. Ask what a patient, capable automated visitor would find, including non-public files that sit behind a public address.
2. Watch for automated probing. Look in your logs for bursts of requests, injection-style parameters and repeated denials. Add rate limits and bot protection to public data services. A run of refusals followed by a change of method is the pattern to look for.
3. Register your own agents. Every agent that can reach outside the organisation needs a named owner, a defined task and an allow-list of where it may connect, enforced in its environment and not only in its instructions. If you cannot list your agents, that is the first finding.
4. Test the stop button. Treat a refusal as a stop, and check that you can switch an agent off quickly, with someone responsible for doing it. A kill switch that has never been tested is a hope, not a control.
5. Publish a monitored security contact. The Medicare notice sat in a general inbox for weeks. A security contact that someone actually reads is cheap, and it is how the next report reaches you in hours instead of weeks.

Questions to ask on Monday

• Which of our systems are reachable from the internet, and which files sit behind them?
• If an automated visitor probed us for a week, would anyone see it?
• Can we list every AI agent we run that can reach outside, and name an owner for each?
• When did we last test that we can switch one off?
• If someone found a problem tonight, who would read their message?

Where to start

Most organisations can do the first two checks with what they already have. The rest usually call for a short readiness check: where your agents run, who owns them, what they can reach and whether the controls exist in practice, not just on paper. The practice runs these, and we are happy to talk it through.

Read more

• The Prime Minister’s disclosure, ABC News
• What is known and not known, ABC News
• OpenAI’s apology and wider disclosures, TechCrunch
• The NSW fire application case, AAP via the Canberra Times
• Parliamentary committee’s early advice, ABC News
• The parliamentary inquiry, Information Age
• Technical reporting, BleepingComputer
• Independent analysis, Cloud Security Alliance
• The sandbox evaluation incident, Malwarebytes