If you work in an office setting, odds are you’ve used generative AI tools to help you do your job at some point.

AI is making its way into every profession. From rushed job applications through to high-profile murder cases in Australia’s top courts (yep, you read that right.)

Legal professionals across the country are being pulled up for using AI tools to prepare important case files. While it might sound like an episode of Black Mirror, it’s happening right now.

In the last month, judges in two separate cases have reprimanded lawyers for submitting court documents filled with evidence and citations that simply… do not exist.

So, how did we get here?

Today, we’ll explore the growing use of AI in the justice system.

Nearly a third of Australian legal professionals use generative AI tools in the workplace, according to a 2024 report from global tech firm Thomson Reuters.

“ Everybody is experimenting with some form of generative AI tool,” UNSW legal expert Professor Michael Legg told TDA.

While publicly available tools like ChatGPT and Claude AI are being put to work, an increasing number of legal firms are paying for access to industry-specific AI programs.

One of the most popular of these is Harvey AI, “ presumably after Harvey Specter from Suits,” Legg said.

There are plenty of use cases for AI in legal practice. These tools can be used to improve efficiencies and run mundane tasks, ultimately freeing up practitioners to focus on other work.

However, one major problem remains: These generative AI models are prone to inaccuracies. They can produce cases, judgments, or quotes that might seem legitimate, but are entirely fake.

In a legal context, this is referred to as an “AI hallucination”.

How often AI “hallucinates” depends on the task and the model. Rates range from as low as 0.6% to as high as 29%, according to AI research company Vectara.

AI hallucinations first emerged in a legal context during a 2023 court case in the U.S. when a man named Roberto Mata attempted to sue Avianca, a Colombian airline.

Mata claims he was injured by a service cart on board a flight. But the case was thrown out when a judge discovered Mata’s lawyers had submitted fake ChatGPT-generated documents to the court.

In the years since, French researcher and legal scholar Damien Charlotin has identified at least 332 cases of AI hallucinations in courtrooms around the world.

While most come from the U.S, Australian courts are not immune from this emerging issue.

At least 23 cases of false, AI-generated submissions have been recorded in Australian courts since 2023.

Last month, lawyers were forced to apologise in the Supreme Court of Victoria after submitting fake quotes and non-existent AI-generated case judgments. The team behind the error were defending a teen on trial for murder.

At the time, Presiding Justice James Elliott said: “It is not acceptable for AI to be used unless the product of that use is independently and thoroughly verified.”

On the other side of the country, a lawyer was fined and referred to WA's Legal Practice Board after submitting ‘AI hallucinations’ in an immigration case.

The lawyer said he used Claude AI as a research tool, and thought the content would be “inherently reliable” after he sought to verify it through Microsoft’s Copilot — using one AI tool to check the work of another.

Justice Arran Gerrard said the incident “demonstrates the inherent dangers [of] solely relying on the use of artificial intelligence in the preparation of court documents.”

Courtroom AI guidelines differ between states.

In NSW, the Supreme Court bans the use of AI to generate certain evidence documents.

Litigants in Victoria are required to disclose if AI was used to prepare their submissions.

And in Queensland, those appearing without a lawyer have been warned that AI is not a substitute for legal advice.

This last point is important because it’s not just lawyers getting in hot water - it’s the people representing themselves in court, too.

The first known instance of unsatisfactory AI-use in an Australian courtroom occurred in 2023, when a WA man representing himself in the state’s Supreme Court used AI to assist his defence.

Since then, 16 other cases of non-lawyers seeking legal counsel from generative AI tools have been recorded in Australian courtrooms.

Some experts argue this practice is a major risk to the legal profession, but this criticism does not address why a litigant might turn to AI for legal assistance.

According to data from LawRight, a community legal organisation, nearly three-quarters of people who self-represent do so because they can’t afford legal representation.

Self-representation is particularly prevalent in the Family Court of Australia, where 39% of trials involved at least one self-represented party between 2019 and 2020.

The complexities of the court system are difficult to navigate without representation, and AI tools are easy and cheap to access.

Professor Legg says AI tools can write very persuasive and believable text, “so people without a legal background are putting their trust in that.”

As AI models become more ingrained in our daily lives, experts suggest AI hallucinations will become more frequent.

At the same time, generative AI could be impacting real-world job opportunities for young people — as AI models are used for tasks traditionally performed by paralegals and junior lawyers.

Data provided to TDA by job search platform GradConnection showed a 6.1% decrease in advertised law graduate roles from pre-COVID levels.

Professor Legg told TDA that people in the justice system perform critical work that AI simply cannot.

While he doesn’t expect AI to “completely replace people” any time soon… On the off chance you find yourself in court soon, it might be worth asking your legal team if and how they use AI.

If you’re dropping $6 a day on café matcha, consider this your sign:

Morning Made’s viral matcha is now at Woolies. Available in Original, Vanilla, and Strawberry, each pouch is made with premium ceremonial grade green tea leaves from Japan – rich in antioxidants, low in sugar, and packed with clean, steady energy.

For one week only, you can snag a pouch for just $8 (normally $10). That’s 10 café-level lattes at home for less than the cost of one. Just add hot water, whisk, pour over milk, and boom – your favourite part of the morning, minus the café queue and price tag.