Five hospitals in the South Metropolitan Health Service of Perth were instructed in May to discontinue the use of ChatGPT for documenting patients' medical records. The Australian Medical Association, in its submission to the government's discussion paper on safe and responsible AI, emphasized the need for robust regulations and transparency in the healthcare industry regarding artificial intelligence. The association highlighted the importance of protecting patients and healthcare professionals, as well as building trust through stronger rules. The decision to cease using ChatGPT for medical notes came after it was discovered that some staff members had been utilizing the language model for this purpose. The CEO of the service, Paul Forden, stated in an email to staff that patient confidentiality could not be ensured with such systems.

The AMA recommended that AI safeguards should include human intervention in clinical decisions, informed consent from patients for AI-based treatments or diagnostics, and protection of patient data to prevent widening health inequalities. The AMA also suggested considering the EU's proposed Artificial Intelligence Act and Canada's requirement for human intervention points in decision-making for Australia. The president of the AMA, Prof Steve Robson, acknowledged the evolving nature of AI and emphasized the need to address the regulatory gap in healthcare to prevent patient harm and privacy breaches. While AI has the potential to improve health outcomes, Google's chief health officer, Dr Karen DeSalvo, stressed the importance of developing appropriate and ethical AI systems. A recent Google research study published in Nature demonstrated that their medical language model provided accurate answers for common medical questions searched online in 92. 9% of cases, on par with clinicians.