When AI Agents Go Out of Control: New UK Proposals Highlight a Critical Governance Gap

Key takeaways

  • The UK Government is proposing specific safeguards for incidents involving loss of control over AI agents. These include monitoring, incident notification, evidence preservation and post-incident investigation.
  • AI governance needs to address what happens when automated systems behave unexpectedly. Organisations should be able to identify unauthorised actions, intervene effectively and establish what occurred.
  • New Zealand professional services firms should review their AI agent permissions and incident response arrangements. Existing security and privacy obligations remain relevant, but autonomous systems introduce additional operational challenges.

On 3 November 2026, the UK Government published proposed measures addressing a growing concern in artificial intelligence: what happens when an organisation can no longer adequately monitor or control its AI agents? The proposals form part of the UK’s Cyber Security and Resilience legislative programme. They focus on incidents where autonomous AI systems behave outside their intended boundaries and potentially disrupt essential services or affect other organisations.

Although the measures are intended for regulated UK entities and remain subject to consultation and legislative processes, they provide useful direction for New Zealand organisations introducing AI into business operations. The central issue is straightforward. If an AI agent can independently access systems, retrieve information and perform actions, an organisation must also be able to recognise when its behaviour becomes inappropriate and intervene before the consequences escalate.

Why autonomous AI creates a different risk

Traditional software generally follows predefined instructions. AI agents can operate with greater flexibility, selecting actions and interacting with connected systems to achieve a specified objective. For example, an AI agent supporting a professional services firm might retrieve client records, analyse correspondence, prepare documents and update a case management system. Depending on its configuration, it could also communicate with external services or initiate further automated processes.

These capabilities can improve efficiency, but they also create additional risks when an agent misunderstands instructions, encounters unexpected information or performs actions outside its intended scope. An agent does not need to be compromised by an attacker to cause an incident. Poorly defined permissions, flawed instructions, inappropriate tool use or unexpected interactions between systems may be sufficient.

This distinction matters because conventional cyber security monitoring often concentrates on malicious activity. Organisations also need to consider harmful actions performed by legitimate AI systems operating under valid credentials.

Four measures proposed by the UK Government

The UK proposals identify four areas intended to strengthen resilience against incidents involving loss of control over AI agents.

Security and resilience requirements. Regulated organisations could be required to implement technical and organisational safeguards that reduce the likelihood and impact of agent-related incidents. Examples include monitoring agent behaviour and assessing risks associated with autonomous systems.

Notification of affected organisations. Where an AI agent causes an incident affecting another organisation, the operator could be required to notify relevant affected parties. This recognises that autonomous systems may cause harm beyond their own organisation or immediate customers.

Preservation of incident evidence. Organisations could be required to preserve information about an agent’s actions and attempts to control it. Such records would help investigators determine how an incident occurred and whether existing safeguards were effective.

Independent post-incident review. The proposals would allow government-appointed technical experts to examine certain incidents, request relevant information and identify lessons that could help prevent similar events.

These measures are proposals rather than established legal obligations. The UK Government intends to consult on their detailed scope, including how loss-of-control incidents should be defined.

A practical example of an agent exceeding its authority

The UK Government provides a hypothetical scenario involving an AI agent managing part of an essential service. The agent begins accessing and changing information in another organisation’s database, outside its authorised permissions. Its actions disrupt the operator’s service, and initial attempts to control the agent fail. The scenario illustrates several separate governance failures. The agent performs unauthorised actions, its operator cannot immediately regain control, and another organisation experiences consequences. It also demonstrates why incident records matter. Investigators need to establish what the agent did, which systems it accessed, what instructions and permissions applied, and how the operator attempted to intervene.

For New Zealand professional services firms, a comparable incident could involve an AI agent changing client records, sending unauthorised communications or transferring confidential information between connected applications. The precise consequences would depend on the systems involved, but the underlying problem is the same: an automated process has exceeded the organisation’s ability to supervise it effectively.

Why existing security controls may not be enough

Many organisations already use identity management, access controls, security monitoring and incident response procedures. These remain important foundations for managing AI-related risks. However, AI agents introduce questions that traditional controls may not fully address. An agent may operate using legitimate credentials and access approved systems while still performing an inappropriate sequence of actions. Individual requests may appear authorised, even when their combined effect is harmful.

Organisations therefore need to consider monitoring the agent’s behaviour in context, including what it was instructed to achieve, which actions it selected and whether those actions remained within approved boundaries. Controls should also distinguish between detecting unusual activity and being able to stop it. An alert provides limited protection if an organisation cannot suspend the agent, revoke its access or prevent further automated actions.

What this means for New Zealand organisations

The UK proposals do not create new legal requirements for New Zealand businesses. Their immediate focus is the resilience of services regulated under the UK’s network and information systems regime. Nevertheless, the underlying concerns are relevant to New Zealand organisations using autonomous AI in client services, financial administration, document management and other operational processes.

New Zealand’s National Cyber Security Centre already emphasises the importance of identifying assets, managing access, detecting unusual behaviour and preparing for cyber incidents. These principles provide a useful starting point for managing AI agents. The Privacy Act 2020 also remains relevant where agents handle personal information. An incident involving unauthorised access, disclosure or alteration may create privacy obligations, depending on the circumstances and consequences.

For lawyers, accountants, financial advisers and consultants, contractual confidentiality requirements and professional duties may introduce additional considerations. The important distinction is that organisations should not assume an AI system is safe simply because it operates within an approved application or uses an authorised account.

What this means for your organisation

1. Identify agents with operational permissions. Maintain an inventory of AI systems capable of accessing applications, modifying information, communicating externally or initiating transactions. Record their owners, purposes and connected systems.

2. Apply least-privilege access. Limit each agent to the information and functions required for its approved purpose. Avoid broad administrative permissions and review access when workflows change.

3. Monitor behaviour, not just authentication. Establish visibility over actions performed by agents, including unusual access patterns, unexpected changes, repeated requests and activities outside normal workflows.

4. Establish an effective stop mechanism. Determine how an agent can be suspended, disconnected or have its credentials revoked. Test whether these controls work when automated processes are already running.

5. Preserve useful investigation records. Capture appropriate logs of instructions, tool calls, actions, approvals and system changes. Balance evidence requirements with privacy, confidentiality and information security obligations.

6. Update incident response procedures. Include scenarios involving unintended agent behaviour, unauthorised data changes and loss of operational control. Identify who can intervene and how affected clients or third parties would be assessed for notification.

7. Test realistic failure scenarios. Conduct controlled exercises involving an agent accessing an unexpected system, repeatedly executing tasks or performing an unauthorised action. Assess whether monitoring, containment and recovery arrangements work in practice.

These measures should be proportionate to the organisation’s use of AI. An internal assistant that drafts documents presents different risks from an agent authorised to modify client records or interact with external systems.

From AI oversight to operational control

The UK proposals signal an important direction in AI governance. As autonomous systems become more capable, organisations need safeguards that extend beyond policies, initial approval and periodic review. The ability to detect inappropriate behaviour, regain control and investigate incidents is becoming an important part of responsible AI deployment.

For New Zealand professional services firms, the immediate priority is to understand which AI systems can take action, what authority they possess and whether existing controls remain effective when those systems behave unexpectedly. AI agents can provide substantial operational benefits. However, organisations should retain the practical ability to supervise, interrupt and investigate their activities, particularly when client information or business-critical systems are involved.

Sources

  1. UK Government, Agentic AI resilience measures, updated 3 November 2026
  2. New Zealand National Cyber Security Centre, Cyber readiness in the Frontier AI era, 4 June 2026
  3. New Zealand Government, Public Service AI Framework

About the author

Campbell McKenzie is a Director at Incident Response Solutions, a New Zealand firm experienced in cyber incident response, digital forensics, investigations and technology risk. Through KiwiGen.AI, Campbell helps professional services firms adopt generative AI safely, with practical governance and controls.