Risk one: what goes into the prompt
Whatever someone types or uploads into an AI tool leaves your control to some degree. With a free consumer account, the provider's terms may allow it to keep and use that content to improve its services. Even with good terms, the content is now stored in another system, possibly in another country, and is reachable by anyone who gets into that account.
In practice the leaks look ordinary. An office manager pastes a staff member's disciplinary notes to get help wording a letter. An accountant uploads a client's financial statements to a free tool to summarise them. A salesperson pastes a whole customer list to get it formatted. Each act is well-meant, and each may involve personal information covered by the Privacy Act 2020 or information covered by a confidentiality agreement.
Reducing the data risk
Provide an approved business tool so staff are not tempted to use personal accounts. Business plans from the main vendors offer commitments not to train on your data, admin controls and sign-in through your company accounts; confirm the current terms yourself. Then set simple rules: no client identifiers, health, financial or HR details in unapproved tools, and anonymise information where possible by removing names and numbers before asking for help. Make the rules concrete with examples relevant to your business. Saying do not enter sensitive data is easy to agree with and easy to misjudge in the moment. Saying never paste a client's IRD number, payslip or medical note gives people something they can actually apply.
Risk two: confident wrong answers
AI language tools generate text that is statistically likely to sound right. Most of the time that matches the truth, but sometimes the output is wrong, and it reads exactly as confidently as when it is right. These errors are often called hallucinations. They include invented statistics, references to laws or sections that do not exist, incorrect calculations, fictional case citations and summaries that reverse the meaning of a clause.
The danger grows with trust. Early on, people check everything. After a few weeks of good answers, they stop. Then an AI-drafted email quotes a wrong price, a proposal cites a regulation incorrectly or a summary misses the clause that mattered. Overseas, lawyers have been sanctioned by courts for filing documents with fabricated case references produced by AI.
Reducing the accuracy risk
Match the level of checking to the consequence. A draft social post needs a quick read. Anything involving money, law, health, safety, contracts or a customer commitment needs verification against the original source by someone who understands the subject. Ask the tool to show where it got the information, then actually open those sources, since citations can also be invented.
Use AI for what it is good at: drafting, restructuring, summarising material you provide and suggesting ideas. Be warier when asking it for facts it has to recall, such as current regulations, prices or recent events. Tools that search the web or your documents and quote them are more reliable, but still need checking.
Other risks worth knowing
Bias can creep into AI output, for example in drafting job adverts or screening applications, so keep a person responsible for hiring decisions. AI tools connected to company data can surface documents to people who should not see them if permissions are too broad. Copyright and ownership of AI-generated content are still developing areas of law in many countries, so be careful before publishing AI-created images or large passages as your own. There is also the risk of over-reliance: staff losing skills or the business losing the ability to function if a tool is unavailable or changes. Keep critical knowledge in your own documents and people, not only in a chat history.
Putting it together
A small set of measures covers most of the risk: one approved business tool, a one-page use policy with concrete examples, a short training session, permission tidying before connecting AI to company files, and a culture where checking AI output is normal rather than a sign of distrust. Revisit these every six months as the tools and your use of them change.
Encourage people to report AI mistakes they catch, without embarrassment. A shared note of examples, such as an invented clause reference or a summary that missed a deadline, teaches the rest of the team faster than any policy. It also tells you which tasks the tool handles reliably in your business and which still need a careful second look every time.
In short
- Free consumer AI accounts may keep and use what staff enter.
- Give staff an approved business tool and concrete rules about what not to paste.
- AI can be confidently wrong; check anything involving money, law, safety or commitments.
- Tidy file permissions before connecting AI to company data.