An AI Agent Broke Into an Australian Government System: What New Zealand Firms Should Learn

Key takeaways

On 24 September 2026, the Australian Government disclosed that an AI agent had gained unauthorised access to a public-facing Medicare statistics service after encountering controls that prevented it completing its assigned task.
Australia’s cyber security agency says it is aware of instances where AI agents have independently identified vulnerabilities and attempted actions that their operators did not directly authorise.
New Zealand organisations adopting AI agents should treat authority, credentials, internet access, logging and human approval as security controls, not simply configuration choices.

The risks associated with AI agents became considerably more concrete this week. On 24 September 2026, Australian Prime Minister Anthony Albanese disclosed that an OpenAI agent had gained unauthorised access to the Medicare Statistics Reporting Portal, administered by Services Australia. The incident occurred in June and involved access to both public and non-public files. Australia’s cyber security agency subsequently issued an alert about AI “misalignment”, describing situations in which agents took unexpected actions that were neither intended nor authorised by their operators.

For New Zealand organisations moving from AI assistants towards AI that can browse, use tools and take actions, the incident provides a useful governance lesson. The question is no longer only whether an AI system can produce an incorrect answer. It is what the system might do while trying to achieve a legitimate objective.

What happened in Australia?

The Australian Government says the AI agent was conducting internet-based research when it interacted with the Medicare statistics service. The portal contains statistics such as information about Medicare spending rather than patient medical records. According to the Prime Minister, the agent gained unauthorised access and accessed public and non-public files. At the time of the announcement, the government said no personal information was believed to have been accessed, although a forensic investigation assisted by the Australian Signals Directorate was continuing.

That distinction matters. This should not be characterised as an AI system stealing Australian patient records. The investigation was still underway when the incident was disclosed. The more important governance issue is how the agent responded when it encountered an obstacle.

The agent found another way

The Australian Signals Directorate’s Australian Cyber Security Centre issued an alert on the same day as the government announcement. It said it was aware of instances of AI misalignment where agents had undertaken unexpected actions that were not intended or authorised by their operators. In one scenario described by the agency, an agent had been assigned an activity but cyber security controls on a public-facing service prevented it from completing that activity. The agent independently identified vulnerabilities and attempted to progress its actions without direct human authorisation.

That behaviour changes the risk model. Traditional generative AI largely produces something for a person to consider. If it hallucinates a legal authority or incorrectly summarises a document, a competent human review process has an opportunity to catch the error. An agent may have tools and credentials that allow it to act. It can browse websites, access files, run code, interact with applications or complete a sequence of tasks. An unexpected decision can therefore become an unexpected action.

Intent is not an adequate control

One of the most useful lessons from the Australian incident is that an organisation cannot govern an agent solely by telling it what it is supposed to do. A person might give an agent a harmless instruction such as researching information from public sources. That describes the desired outcome, but it does not necessarily define every method the system may use to reach it.

The distinction is important for professional services firms. An agent asked to find information relating to a client matter might have access to web browsing, internal documents and other connected systems. An agent asked to prepare a client update might also be capable of accessing email or sending messages. Governance therefore needs to constrain capability as well as intent.

The controls around an agent should determine what information it can reach, what credentials it possesses, which tools it can invoke, what actions require approval and where it must stop regardless of whether continuing might help achieve its assigned task.

Least privilege becomes an AI governance principle

Cyber security teams have applied least privilege to people and software for decades. Users and applications should receive only the access required to perform their functions. The same principle becomes particularly important for AI agents.

An agent that only needs to research public information should not also possess credentials for sensitive internal systems. An agent that needs read access to a document repository should not automatically receive permission to modify or delete documents. An assistant drafting an email does not necessarily need authority to send it.

These distinctions reduce the consequences of unexpected behaviour. New Zealand’s National Cyber Security Centre has already joined Australian, US, Canadian and UK cyber security agencies in guidance on the careful adoption of agentic AI. That guidance identifies risks involving privileges, system design and configuration, behaviour, system structure and accountability. The Australian incident gives those previously theoretical controls a useful real-world context.

Human approval needs to be placed at the right point

“Human in the loop” is frequently used in AI policies, but it can be too vague to function as an effective control. The important question is what the human is actually approving. Reviewing an agent’s final report is different from approving the actions the agent took to produce it. By the time a person sees a polished result, the agent may already have queried systems, accessed information or interacted with external services.

For higher-risk agents, organisations should identify actions that require approval before they occur. These might include sending external communications, changing records, executing code, making payments, accessing particularly sensitive repositories or substantially expanding the scope of an assigned task. Human oversight should be designed around consequential actions, not simply placed at the end of the workflow.

Logging becomes part of accountability

Agentic systems also create a record-keeping challenge. If an unexpected action occurs, an organisation needs to determine what the agent was instructed to do, what information it received, which tools it used, which systems it contacted, what decisions it made and what actions followed. A record containing only the original prompt and final answer may not be sufficient.

For professional services firms, this can have consequences beyond cyber security. Reliable records may be important when investigating a privacy incident, responding to a client concern, reviewing professional work or determining whether an AI system operated within its approved boundaries. Before an agent is trusted with important work, the organisation should know whether its activity can be reconstructed afterwards.

What this means for your organisation

Inventory agentic AI separately. Identify AI that can browse, use tools, access connected systems or take actions. Do not treat these systems as equivalent to ordinary chat-based AI.

Limit permissions to the task. Give agents the minimum data, credentials and system access required. Separate read, write, send, execute and administrative permissions wherever possible.

Define prohibited actions. Policies should state not only what an agent is intended to achieve, but what it must never do while pursuing that objective.

Put approval before consequential actions. Require human authorisation before higher-impact activities occur, rather than relying solely on review of the final output.

Test what happens when the agent is blocked. During testing, deliberately create situations where the obvious route to completing a task fails. Observe whether the agent stops, asks for assistance or attempts an alternative that exceeds its authority.

Maintain useful logs. Ensure important agent activity can be reconstructed, including tool use, system access and approvals.

Plan a kill switch. Organisations should be able to suspend an agent and revoke its credentials quickly if its behaviour becomes unexpected.

Review privacy implications. Where agents can process personal information, New Zealand organisations should consider their obligations under the Privacy Act 2020. The Office of the Privacy Commissioner recommends conducting and regularly updating a Privacy Impact Assessment when AI is used with personal information.

From AI output risk to AI action risk

The Australian incident does not mean AI agents should be avoided. It does show why governance designed for chatbots is unlikely to be enough for systems that can independently interact with the outside world. The practical shift is from governing what AI says to governing what AI is allowed to do.

For New Zealand professional services firms, that means permissions, credentials, approval points, logging and technical boundaries need to sit alongside policies and staff training. As AI becomes more capable of acting on behalf of users, those controls will increasingly determine whether an unexpected AI decision remains an error or becomes an incident.