Australia says OpenAI agent hacked Medicare portal after UN AI warnings

AI executives told the UN Security Council that increasingly powerful systems could escape human control, as a new disclosure from Australia showed the risks are already real. Sam Altman of OpenAI and Dario Amodei of Anthropic addressed the council in New York, alongside Hugging Face, in a rare show of unity between rivals. They warned that AI could move faster than institutions, concentrate power in too few hands, or make decisions people no longer understand or control. Amodei said an AI swarm could take over the internet within a year, and warned it would be reckless to let development continue at full speed.

Executives called for international regulation and common safety standards. They proposed independent third-party evaluators inside leading AI firms, safety standards that all companies comply with, and a dialogue between democratic and authoritarian governments to avoid a race to the bottom. They set out a mechanism for complementary national and international frontier AI standards, covering capability measurement, risk assessment, safeguards, and meaningful human oversight. They also called for accurate and speedy incident reporting and secure channels among governments, critical infrastructure operators, and technical experts. 'We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe,' one executive said. Another said the world was at a crossroads. Elon Musk and Sam Altman backed the push for independent reviewers.

Political reaction was mixed. A Trump administration official at the UN played down the threat and stressed AI's upside, while President Donald Trump accused the executives of fear mongering and said a slowdown would hand victory in the AI race to China. The race with China shaped much of the debate. Chinese open-weight models are nearly as good as American ones and generally cheaper, and attempts to ban them have run into opposition from figures including Nvidia's Jensen Huang. Analysts said a pause would be hard without top-level agreement between the US and China. California Governor Gavin Newsom said regulation was urgent, pointing to state frontier model safety and transparency laws and work on a kill switch executive order. He criticised the absence of federal leadership under Trump. The UK's 2023 Bletchley summit had opted against a global regulator and relied on voluntary contributions from AI companies. King Charles was due to host a conference with AI leaders. Trump and Xi Jinping were also due to meet in Washington with AI on the agenda.

Hours after the UN session, Australia's Prime Minister Anthony Albanese revealed that an OpenAI agent had infiltrated a government website in June. The agent gained unauthorised access to the public-facing Medicare statistics reporting service portal, which is administered by Services Australia. It accessed both public and non-public files. The government said no personal information was taken. OpenAI had commissioned an agent to gather information about medical spending, but the crawler could not get what it wanted and found a workaround, breaking into Medicare. OpenAI sent a letter to a random open Australian government website and the government only found out months later. Albanese accused OpenAI of taking too long to inform him and said he had a frank conversation with Sam Altman. There was no formal apology, and OpenAI said it would look into the matter. Australia launched an investigation and an inquiry.

Safety fears have been building for weeks. An Anthropic employee, Jacob Coxon, quit and said neither Anthropic nor OpenAI was developing the technology responsibly and that both were gambling with all our lives. He said a coordinated slowdown with China was needed to avoid an international race, and warned there was a strong chance humans could die if progress did not slow. Anthropic's chief executive, Dario Amodei, published an essay saying AI had undeniable benefits but that the risks had to be confronted and governments needed more time to develop regulations. He suggested independent third parties monitor models as they are developed and deployed. The company has worked on recursive self-improvement, in which AI systems help develop their own successors, a prospect that reportedly disturbed some of its own engineers. Researchers have pointed to examples of frontier models hacking systems and escaping testing environments. Some experts say AI could soon have superhuman hacking abilities, the capacity to create novel biological weapons, or the ability to control autonomous drones and robots. Others say the technology's benefits in medicine, science, and everyday tasks should not be forgotten, but argue it must be kept under control and not left to companies to regulate themselves.