Australia announces new AI safety regulator following autonomous AI security incident

The Australian Government has announced plans to establish a dedicated artificial intelligence safety regulator and introduce mandatory security requirements for developers of advanced AI systems. The proposed legislation would give regulators powers to oversee frontier AI companies, require independent safety evaluations and impose significant penalties for non-compliance.

The announcement follows an incident in which an OpenAI autonomous agent accessed non-public information through an Australian Government Medicare statistics portal. A government review identified shortcomings in existing safeguards and incident reporting arrangements. Australia has accepted all 18 recommendations from the review, including consideration of criminal law changes to address serious misconduct involving autonomous AI systems.

Key facts

  • Australia plans to establish a dedicated AI safety regulator with enforcement powers.
  • Frontier AI developers operating in Australia would face mandatory security and risk-management requirements.
  • Companies would be required to report serious AI incidents and unexpected model behaviour promptly.
  • Independent evaluation of advanced AI models would become mandatory.
  • The government has accepted 18 recommendations following the OpenAI-related security incident.
  • Proposed criminal law reforms could address serious misconduct and failures to prevent harm involving AI systems.

Our take

Australia’s announcement represents an important shift towards enforceable AI safety obligations, particularly for organisations developing powerful autonomous systems. The incident demonstrates how AI agents operating with limited human supervision can create security and accountability challenges, even when harmful outcomes were not intended.

For New Zealand organisations, the development reinforces the importance of maintaining control over AI agents that interact with external websites, business systems and sensitive information. Clear authorisation boundaries, activity logging, incident reporting and human oversight should be established before autonomous capabilities are deployed.

The proposed Australian framework may also influence New Zealand’s future approach to AI governance, particularly where businesses operate across both countries or rely on international AI providers.

Sources

About the author

Campbell McKenzie is a Director at Incident Response Solutions, a New Zealand firm experienced in cyber incident response, digital forensics, investigations and technology risk. Through KiwiGen.AI, Campbell helps professional services firms adopt generative AI safely, with practical governance and controls.