Maskot/Getty ImagesVisits to a new doctor, physio or specialist often begin with: “Do you mind if I use AI?”According to Off the Record, a new report by Digital Rights Watch, more than 40% of Australian doctors are now using AI scribes. These tools listen to consultations, feeding what they hear into Large Language Models – similar to ChatGPT or Claude – to generate medical notes.For clinicians, AI scribes are an easy sell. They eliminate hours of paperwork, freeing energy to focus on the patient. This can profoundly change how chronically burned out doctors feel about their work. And with Medicare only compensating doctors for face-to-face time, and not note-taking, the financial and emotional incentives for practitioners are undeniable.Yet the benefit for patients is less clear, especially when we have little transparency about who has access to deeply personal medical data, or how this data is handled by these lightly regulated AI scribes. How do AI scribes work? And what can go wrong?Many AI scribes don’t just transcribe text. They summarise and interpret conversations based on what the AI assumes a medical professional might say.However, AIs often hallucinate, confidently stating things that never happened. A doctor counselling a patient to quit smoking might find the AI confidently recorded that the patient was advised to avoid house fires.AI systems also frequently show biases relating to race, ethnicity, class and gender. For patients from culturally and linguistically diverse backgrounds, or those for whom English is a second language, AI interpretations of speech can lead to medical records that wildly diverge from reality.Early studies suggested up to 90% of AI-generated notes required correction. More recent studies show that approximately 20% of AI clinical notes contain errors significant enough to impact patient diagnoses. And catching these errors is hard for clinicians, as humans really struggle to identify others’ mistakes under time pressure. This is especially true when automation bias leads humans to accept an AI’s output without scrutiny. But failing to correct these errors harms both patients and any subsequent clinician relying on the notes. We’ve seen this in the legal space, where judges and lawyers are now forced to meticulously fact-check every submission to ensure no AI hallucinations slip through, slowing down proceedings. If clinicians must meticulously verify every note for potential AI-related mistakes, the promised productivity gains likely evaporate. If they don’t, the health system will be flooded with unreliable data. Clinicians aren’t well-equippedThe issues with trust run much deeper. Medical professionals aren’t well-equipped to provide guidance on where the data they give to AI scribes goes. At the moment, patients are asked just to trust that their clinician understands what is happening. But health-care professionals aren’t AI experts and aren’t well equipped to discuss these privacy issues. Their professional bodies provide only high-level information. Complicating this is the fact AI scribes typically fail to disclose how data is handled, who has access to it, and what AI models are used. This means that beyond vague assurances that data remains in Australia, health-care professionals simply cannot give patients concrete answers about who is accessing their data or what they are doing with it.What needs to happen?Regulators are struggling to keep pace: they’re working with laws that don’t clearly define who is responsible for this rapidly evolving space.As Digital Rights Watch argues, Australia’s Therapeutic Goods Administration (TGA) should regulate AI scribes as it would any other medical device. It should be subject to the same safety testing requirements as any other piece of medical equipment.The TGA currently claims scribes are only medical devices if they provide diagnostic advice. This allows vendors to evade scrutiny. The TGA’s position ignores how generative AI is used in clinical practice, and that even transcriptions can contain hallucinations or biases that could affect diagnoses. This leaves Australia’s guidance out of step with nations such as the United Kingdom.Fundamentally, the government must ensure Australians’ private data is properly safeguarded, through both legislation and bodies such as the AI Safety Institute. This should include forcing AI service providers to clearly disclose what data they have access to, how they use it, and who can access it, with rigorous protections to ensure privacy and real penalties for breaches. More broadly, there is a need for rigorous independent testing to ensure the AI systems with which we interact are fair, reasonable and unbiased for all Australians. This should include testing whether these systems produce genuine productivity gains that are beneficial for patients. Without these safeguards, Australian patients may lose trust in the medical system. And once that trust is broken, it will hurt us all.Andrew Cullen is a board member of Music Technology Australia, a member of the World Intellectual Property Organisation's AI Technical Exchange, and manages the consultancy EthiCan.