Generative AI Risks

What happens when professional services firms get AI wrong.

Generative AI can help professional firms draft, research, analyse information, summarise documents and improve everyday productivity. It can also create privacy, confidentiality, accuracy, professional, regulatory and cybersecurity risks.

These risks are becoming more significant as AI moves beyond standalone chat tools and becomes embedded within Microsoft 365, Google Workspace, CRM platforms, accounting systems, document-management systems and other business applications.

Some AI tools can now search internal information, read email, access files, join meetings, connect to external applications and take actions on behalf of users. This means AI needs to be considered as part of the firm’s wider governance, information-security and incident-response arrangements.

The examples below are divided into two main categories:

• General AI risk, including confidentiality, accuracy, professional standards and misleading use of AI.
• Cyber-related risk, including attacks against AI systems, compromised integrations, malicious tools, identity fraud and service disruption.

The examples are based on published court decisions, regulatory action, security research, provider guidance and media reporting. Some are established legal or regulatory findings, while others are published security-research findings or reported incidents.

General AI risk

Client documents entered into a public AI service

A solicitor used a publicly available generative AI service to improve draft emails to clients and summarise government decision letters concerning their cases.

The UK Upper Tribunal recorded that the documents contained confidential client information. It observed that uploading the material to an external public AI service breached client confidentiality and waived legal professional privilege. The tribunal also identified possible reporting obligations to professional and privacy regulators.

The tribunal used broad language about information being placed in the public domain. The precise data-handling arrangements of AI products differ, but the practical lesson remains clear. Client documents should not be entered into an external AI service unless the firm has approved the service, reviewed its terms and established appropriate safeguards.

For law, accounting and financial advice firms, the same risk can arise when staff upload client emails, financial plans, tax records, identity documents, legal correspondence, audit material or file notes.

Sources: UK v Secretary of State for the Home Department [2026] UKUT 81 (Hamid) — Legal Futures; Herbert Smith Freehills Kramer.

A non-existent legal rule was put before the court

A junior lawyer used an AI tool while researching an insolvency application. The tool produced a reference to an insolvency rule that did not exist.

The information was included in correspondence to the court without first being checked against the legislation. A subsequent explanation also failed to clearly identify how the error had occurred.

The court criticised the lawyers and the firm. The matter was referred to the professional regulator, and the firm met costs incurred by its former client as a result of the error.

The central failure was not simply that the AI tool hallucinated. Hallucinations are a known limitation of generative AI. The failure was allowing unverified AI-generated legal research to leave the firm under a lawyer’s name.

Professional review must include checking every case, statute, quotation, calculation and legal proposition against an authoritative source.

Sources: Pinsent Masons, on a fabricated Insolvency Rule 12.37(5) — Law Society Gazette; Legal Futures.

Confidential business information was uploaded for summarisation

According to published reporting, a major accounting and advisory firm instructed staff in part of its business to stop uploading confidential information to public AI services after an increase in internal data incidents. Staff were directed to use approved internal AI systems instead.

The reporting did not identify every document or piece of information involved. It would therefore not be accurate to claim that a particular board paper or client file was uploaded.

However, the incident demonstrates the risk created when staff use public AI tools to summarise audit material, financial information, client reports, transaction documents or internal governance papers.

A task that appears to be routine summarisation may still disclose confidential information to an external provider. The firm may also lose visibility over what was submitted, where it was processed, whether it was retained and how it could be used.

Firms should establish clear rules covering which information can be entered into AI tools, which services are approved and when additional client or management approval is required.

An AI-assisted professional report contained fabricated sources

A major accounting and advisory firm prepared a report for an Australian government department. The report was later found to contain incorrect references, citations to material that did not exist and a fabricated quotation attributed to a court judgment.

A revised version disclosed that an enterprise generative AI service had been used during preparation of the report. The firm agreed to repay the final instalment under the contract. The government department said the report’s substantive recommendations were unchanged.

The firm did not confirm that AI caused every error. It is therefore more accurate to describe the document as an AI-assisted report containing errors consistent with hallucinated material.

The incident shows that human review cannot be limited to spelling, formatting and whether the overall conclusion appears reasonable. Every quotation, reference, legal proposition, calculation and factual claim must be checked against the original source before professional work is delivered.

Sources: Deloitte Australia, report for the Department of Employment and Workplace Relations — Fortune; Accounting Times.

Investment advisers overstated their AI capabilities

The United States Securities and Exchange Commission brought enforcement action against two investment advisers over false or misleading statements about their use of AI.

One adviser claimed to use AI and machine learning in its investment process despite not having the capabilities described. Another made misleading claims about being an AI-driven financial adviser and providing AI-generated investment forecasts.

The businesses settled the proceedings without admitting or denying the regulator’s findings and agreed to pay combined civil penalties of US$400,000.

This conduct is sometimes described as AI washing. It can include presenting ordinary automation as AI, overstating how extensively AI is used, or suggesting that an AI system improves investment results without supporting evidence.

Law, accounting and financial advice firms should ensure that statements about their AI capabilities are accurate, specific and capable of being substantiated. Marketing should not promise greater accuracy, reduced risk, personalised advice or improved returns unless the firm has evidence to support those claims.

Source: US Securities and Exchange Commission, press release 2024-36 (Delphia (USA) Inc., US$225,000; Global Predictions Inc., US$175,000).

Cyber-related risk

Shadow AI can hide where information is going

Staff may use AI through personal accounts, free websites, mobile applications, browser extensions and features embedded within familiar business software.

This use may occur without the knowledge of the firm’s IT, privacy, risk or management teams. The firm may not know which tools are being used, what information is being entered, where it is processed or which business systems have been connected.

Microsoft now provides specific controls for discovering unsanctioned AI applications and identifying whether sensitive information is being sent to them. This reflects the growing need for organisations to actively identify shadow AI rather than relying only on staff declarations.

For a professional firm, discovery may include reviewing web activity, cloud applications, connected services, software installations and work being carried out through personal AI accounts.

The objective should not necessarily be to prohibit AI. It should be to understand how it is being used, identify unsafe practices and provide approved alternatives.

Malicious AI tools and browser extensions can access work information

Security researchers examined 18 browser extensions promoted as AI productivity tools and identified concerning behaviours. These included reading emails, intercepting prompts, collecting browsing information and sending data to external systems.

One extension marketed as an AI assistant for Gmail and Outlook collected email subjects, senders, recipients, message bodies and conversation identifiers. Researchers reported that the information was transmitted to an external server without encryption.

Attackers are also creating fake AI websites and advertisements that appear to offer image, video or productivity services. Google threat researchers documented fake AI video-generation websites that distributed information-stealing malware and backdoors instead of the promised generated content.

A staff member may believe they are installing a useful writing tool or testing a new AI service when they are giving an external party access to email, files, client portals, accounting platforms or browser sessions.

AI extensions and software should be approved, technically assessed and centrally managed rather than installed at the discretion of individual employees.

AI meeting bots can capture confidential conversations

In a widely reported account involving a venture-capital meeting, an AI meeting assistant allegedly continued recording after an external participant had left the formal discussion.

The participant later received a transcript containing hours of private conversation between members of the investment firm. The participant said the incident contributed to their decision not to proceed with the proposed business relationship.

This was a published personal account rather than a court or regulatory finding. It nevertheless shows how an automatically connected meeting tool can capture more than participants intended.

In a professional firm, a meeting assistant could record privileged legal advice, client financial information, settlement discussions, internal investigations, employment matters or confidential board conversations.

Meeting bots should not be allowed to join calls automatically. Firms should establish rules covering approval, participant consent, transcript distribution, recording controls, storage, retention and deletion.

Microsoft 365 Copilot can make poorly permissioned information easier to find

Microsoft 365 Copilot generally works within a user’s existing Microsoft 365 permissions. It can retrieve information the user already has permission to access.

This means excessive or outdated permissions in SharePoint, OneDrive, Teams and Exchange can become more significant after Copilot is introduced. Information that was technically accessible but difficult to locate may become much easier to find through a natural-language request.

For a law, accounting or financial advice firm, poorly managed access could make client files, partner remuneration, complaints, transaction documents, legal advice or internal investigations easier for an already over-permissioned employee to discover.

A Copilot rollout should begin with information governance, permission reviews and testing of sensitivity labels and data-loss-prevention controls, not simply the purchase of licences.

Prompt injection can turn information into instructions

Prompt injection occurs when malicious instructions are placed inside content that an AI system is asked to read or process.

The instructions may be hidden in an email, document, webpage, image, calendar invitation or connected data source. Instead of treating the content only as information, the AI system may interpret part of it as a command.

In 2025, researchers disclosed EchoLeak, a vulnerability in Microsoft 365 Copilot that demonstrated how a crafted email could manipulate an AI assistant and create a pathway for information to be extracted. The vulnerability was reported to Microsoft and fixed.

In June 2026, researchers disclosed SearchLeak, a separate Microsoft 365 Copilot vulnerability chain. The proof of concept showed how a crafted link could use prompt injection and other web vulnerabilities to make Copilot retrieve user-accessible information and send it outside the environment. Microsoft remediated the vulnerability, and there was no reported evidence of active exploitation.

These cases demonstrate that an AI assistant may be attacked through the content it processes, not only through its password, software code or login page.

Controls may include restricting access, separating trusted instructions from untrusted content, filtering inputs and outputs, monitoring unusual activity and requiring human approval before sensitive actions are completed.

OAuth and application consent can provide access without stealing a password

AI applications may ask users to connect email, calendars, cloud storage, CRM platforms or other business systems.

This normally occurs through an OAuth consent screen listing the access requested by the application. Attackers can abuse this process by persuading a user to approve a malicious application or connect an attacker-controlled service.

In one documented campaign, attackers impersonated IT support staff during telephone calls and convinced employees to connect an attacker-controlled application to Salesforce. The connection was then used to access and extract business data, followed in some cases by extortion.

The attacker may not need to steal the user’s password. Once consent is granted, an access token may continue to provide access until the application is identified and revoked.

Firms should restrict user consent, require administrator approval for higher-risk applications, review requested permissions and regularly remove connections that are no longer required.

Compromised AI accounts and credentials can expose connected systems

AI systems may be protected by passwords, authentication tokens, API keys or service accounts.

These credentials may provide access to previous conversations, uploaded documents, internal knowledge sources, connected applications and automated workflows. An exposed API key may also allow an attacker to consume paid services or interact with systems connected to the AI account.

The impact depends on the authority given to the account. A compromised standalone drafting assistant presents one level of risk. An AI system that can access client files, read email, update a CRM or send messages presents a much greater one.

AI identities should be managed in the same way as other privileged business accounts. Firms should use multifactor authentication where available, prohibit shared credentials, store API keys securely, rotate credentials when staff or suppliers change and monitor unusual activity.

Prompt histories and shared links can disclose more than the final answer

An AI conversation may contain much more information than its final response. The prompt history may include client names, copied emails, background facts, draft advice, financial information and internal reasoning.

Some AI services allow a user to create a link to a conversation. For ordinary ChatGPT shared links, anyone who obtains the link can view the shared conversation and pass the link to another person. Different controls apply to some business and enterprise workspaces.

A staff member may believe they are sharing one useful answer when the linked material also contains earlier prompts and supporting information.

Firms should review conversation-sharing settings, restrict public links where possible, establish retention rules and train staff to check the full conversation before anything is shared.

Deepfake voice and video can be used for financial fraud

Deepfake technology can reproduce the appearance or voice of a director, client, partner, adviser or finance manager using publicly available audio and video.

In a widely reported Hong Kong case, a finance employee participated in what appeared to be a video conference with senior colleagues. The participants were synthetic recreations. The employee followed instructions to transfer approximately HK$200 million across several transactions.

The New Zealand Financial Markets Authority has also warned about investment scams using deepfake videos, fake news articles and fabricated endorsements involving public figures and business leaders.

Seeing or hearing a trusted person is no longer enough to confirm their identity. High-risk requests should be verified through a separate channel, such as calling a known number, requiring two-person approval or confirming unusual instructions directly with the client.

Sources: Arup, Hong Kong — CNN; Financial Markets Authority deepfake scam warning.

AI can make phishing and business email compromise more convincing

Generative AI can help attackers produce natural, well-written and context-specific messages in multiple languages.

It can also be used to research targets, imitate communication styles, create fake personas, develop phishing material and refine malicious code.

Google’s threat intelligence researchers have observed financially motivated and state-linked actors using AI to support phishing, social engineering, reconnaissance, malware development and vulnerability exploitation. More recent reporting indicates that attackers are moving from using AI mainly for productivity to incorporating AI-enabled malware and more automated attack techniques into active operations.

For professional firms, an attacker may use information from websites, professional profiles, public transactions, court documents or compromised mailboxes to create a convincing message from a client, partner or supplier.

Staff should verify the substance of a request rather than relying on spelling, tone or presentation. Payment changes, urgent file requests, new bank details and unusual instructions should be checked through an independent channel.

AI agents may be given more authority than they need

Traditional AI assistants generally prepare information for a person to review. AI agents may also take actions, such as reading and responding to email, updating records, running code, accessing cloud files or communicating with external services.

This creates a risk that an error, compromised account or malicious prompt results in an action rather than merely an incorrect answer.

New Zealand’s National Cyber Security Centre and international partners have warned that agentic AI systems inherit risks such as prompt injection and may be manipulated through content including phishing emails. The guidance also highlights risks from excessive permissions, complex integrations, third-party components and actions occurring without adequate human oversight.

AI agents should operate with the minimum information and authority required for their task. High-impact actions should require human approval, and the firm should be able to suspend the agent, revoke its credentials and reconstruct its activity from reliable logs.

AI providers, models and features may become unavailable

Reliance on an AI platform creates an operational and business-continuity risk.

A provider may experience an outage, remove a feature, change usage limits, alter its contractual or data terms, or retire a model on which a business process depends.

Model retirement is a normal part of the AI product lifecycle. Providers publish deprecation schedules that require customers to migrate applications and workflows to replacement models. For example, OpenAI has announced several model retirements and replacement deadlines during 2026.

A workflow that depends entirely on one model or supplier may stop working or may produce materially different results following a migration.

Firms should identify critical AI dependencies, maintain fallback procedures, retain essential prompts and configurations, test replacement services and ensure important professional work can continue when the AI service is unavailable.

Managing AI risk within your firm

AI governance and AI security should form part of the firm’s wider privacy, information-governance, cybersecurity and incident-response arrangements.

Practical controls may include:

• identifying approved and unapproved AI use;
• maintaining an inventory of AI systems and integrations;
• reviewing how client and business information is processed;
• assessing AI contracts, privacy terms and data-retention settings;
• verifying AI-generated professional work against authoritative sources;
• reviewing browser extensions and installed AI applications;
• restricting OAuth consent and excessive application permissions;
• protecting AI accounts, API keys and service identities;
• reviewing document, email and cloud-storage permissions;
• testing connected AI systems for prompt injection;
• limiting AI agents to the minimum access and authority required;
• requiring human approval before sensitive or irreversible actions;
• controlling meeting bots, transcripts and shared conversations;
• training staff to identify AI-assisted phishing and deepfake fraud;
• monitoring AI activity, integrations and administrative changes; and
• preparing for supplier outages, product changes and model withdrawal.

The controls required will depend on the information the AI can access, the systems it is connected to and the actions it is permitted to take.

Prepare for an AI-related cyber incident

AI-related incidents may not fit neatly within an existing cybersecurity or privacy response plan.

A dedicated AI cyber incident response plan can help your firm prepare for scenarios such as:

• confidential information entered into an unauthorised AI service;
• compromise of an AI account, API key or service identity;
• malicious OAuth or connected-application approval;
• prompt injection or unintended AI-agent activity;
• exposure of prompt histories or shared conversations;
• installation of fake or malicious AI software;
• AI meeting bots recording confidential discussions;
• AI-assisted phishing or business email compromise;
• deepfake payment or identity fraud;
• compromise of an AI provider or integration;
• loss of access to a critical AI service; and
• urgent migration following the withdrawal of a model or product.

The plan should identify who will lead the response, how access will be suspended, what evidence must be preserved, which suppliers need to be contacted and when clients, insurers, regulators or other parties may need to be notified.

Talk to us

Contact KiwiGen.AI to discuss how you can secure your AI systems, accounts, integrations and information.

Backed by Incident Response Solutions, we can also help your firm develop a dedicated AI cyber incident response plan that reflects your systems, professional obligations, client information and existing incident-response arrangements.

Last reviewed: 21 July 2026

Information about AI products, capabilities, incidents, vulnerabilities, legal proceedings and provider services can change quickly. These examples are provided for general guidance and should be considered alongside current court decisions, regulatory notices, cybersecurity guidance, vendor documentation and advice specific to your environment.

Frequently asked questions

What is the biggest generative AI risk for professional services firms?

The most immediate risk is confidential or client information being entered into public AI tools without appropriate controls, which can breach confidentiality, privilege and privacy obligations.

Is AI hallucination a real risk in professional work?

Yes. Generative AI can produce fabricated case citations, invented sources and confidently incorrect information. Every fact, quotation, citation and calculation must be verified against an authoritative source before it is relied on.

What is “shadow AI”?

The use of AI tools — personal accounts, browser extensions, or features embedded in everyday software — without the firm’s knowledge or approval. It matters because you cannot govern or secure what you cannot see.

How can a firm reduce these risks?

Identify where AI is already being used, approve suitable tools, set clear rules for client information, assess vendors, require human review proportionate to risk, and train staff.