A Response To The Australian Government's Response To The Senate AI Report
The Australian Government continues to ignore the safety risks from advanced AI.
Below is a copy of a letter I wrote in response to the Australian Government’s response (April 2026) to the The Senate Select Committee on Adopting Artificial Intelligence (AI) (November 2024).
The TLDR is — the government is ignoring physical safety risks posed by AI, and moving far too slowly to be able to process the developments in the field. I hope this gets better with the establishment of the AISI, but do remain concerned.
To Whom It May Concern,
My name is Hunter Jay, I am a software engineer, and the director of the Sydney AI Safety Space. I have been focused on AI and AI safety for nearly 10 years, and have notably accurate predictions on AI progress -- for example, ranking in the top 2% of forecasters in a recent competition.
I am writing to express some disappointment with the government’s response to the Select Committee on Adopting Artificial Intelligence, to which I wrote a short submission in early 2024. I believe the government response ignores the two most important risk categories of this technology.
While I am very glad that part of my recommendation -- to establish an Australian AI Safety Institute (AISI) -- is being acted on, I overall remain concerned that the government does not understand the risks we face with artificial intelligence, nor the speed at which those risks are likely to eventuate, and is so far failing to act to protect Australians.
There are two main categories that were not addressed in the response, which I would like to bring the government’s attention to.
Misuse of Highly Capable AI Models
As AI capabilities increase, and especially as open weight models of higher capability become available, the difficulty of taking destructive actions such as creating infectious diseases or running cyberattacks on critical infrastructure lowers, such that the number of people or adversarial groups capable of taking such actions increases significantly every few months.
In the worst case, an engineered pathogen could cause a pandemic significantly worse than COVID-19, as the disease can be modified to be more infectious and more deadly (or potentially selectively deadly, targeting specific ethnic groups). AI enabled biorisks are now being documented by the Center for Strategic and International Studies, the Nuclear Threat Initiative’s December 2025 report ‘Safeguarding Against Global Catastrophe‘, and research published in Science in February 2026, but are not yet addressed by the Australian government as far as I am aware. Fortunately, biorisks have not yet eventuated, and we may have several months before AI models reach the level where this danger is serious.
On the other hand, AI assisted cyberattacks are already happening, and the most recent AI models are capable of finding high-severity vulnerabilities at scale.
We are fortunate that Anthropic, the company behind the current most capable models, is cautious about these risks and is therefore including protections, and working on a sensible release schedule by providing their tools to cyberdefenders first, in an effort to patch the vulnerabilities that can be found automatically. That said, it is a failure of governments worldwide that we are relying on the good behaviour and voluntary, costly actions taken by private companies for our security.
Loss of Control of Extremely Capable AI Models
In the slightly longer timeframe -- potentially in two to five years -- we face the risk that AI models themselves become capable of evading efforts to control or oversee them, and therefore the values encoded in those AIs (intentionally or not) will determine what the future looks like far more than humans do. It will be these AIs building future AIs, developing technology, doing work, and generally acting in the world. The arrival of AIs that are more capable than humans on this timeframe is the default expectation of many mainstream figures in the space, including the CEOs of all three leading frontier labs.
The government response does not engage with this at all, with no discussion of the problem of how to align the AIs to moral values at a technical or policy level, nor about how failing to do this would necessitate a pause in AI capability development.
It is also notable that the same day this response was released, Dario Amodei of Anthropic was meeting with the government in Canberra raising the same issues I am raising here, quoted in Capital Brief as saying “The cyber risks are serious. The bio risks are serious.”
While I appreciate this response to a report about the adoption of AI, I think the physical safety of people should be in consideration when discussing government involvement in technology adoption. A response to a report on the adoption of nuclear energy, for instance, would rightly include a discussion of physical safety risks, even though those risks are far better understood, and far smaller in scale, than the risks posed by advanced AI.
I also appreciate that the upcoming AISI will help the government to understand these risks, and I hope that it is able to move at the pace the technology demands. That said, the slow speed of its establishment and its small budget do not seem to be commensurate with addressing risks of the magnitude discussed above. I urge the government to move faster here, and to expand the AISI beyond evaluations and communication into actually doing the technical research and policy work needed to address these risks.
Thank you,
Hunter Jay