Dear Senators, AI is an existential risk
This was my May 2024 letter to the Select Committee on Adopting Artificial Intelligence (AI), from the Australian Senate.
This was my May 2024 letter to the Select Committee on Adopting Artificial Intelligence (AI), from the Australian Senate. They asked for submissions from individuals and groups about their views on adopting AI, and on issues and how they could be addressed. You can find all the public submissions here.
Dear Senators,
My name is Hunter Jay. I am a software engineer and AI developer, and co-founded the startup Ripe Robotics, which works on automating apple and stone fruit harvesting.
I am worried that the gravity of artificial intelligence development is being understated. Many underplay it because it seems fantastical, and others underplay it because they don’t want to seem weird.
So, I’ll try to state as clearly as I can here -- if current trends continue, artificial intelligence could kill literally everybody on earth before you campaign for re-election in 2031.
This is not a fringe view (although researchers’ estimates and timeframes vary).
The current generation of A.I. is a tool that can make people more productive (to positive or negative ends) and should be treated and regulated as such. The next generation will likely be the same. There is no existential risk from current AI systems.
The problem comes after that, when we have an AI which is more capable than a person at nearly every task, which is able to make copies of itself, and whose capabilities grow exponentially without human input. Creating AGI is the explicit goal of OpenAI, Anthropic, and Google/Deepmind, and it is plausible that systems like this could be built this decade1.
If such an AI is built, it will have control of the future, not us.
That could be an extremely good thing. Imagine the equivalent of 10 billion researchers working in perfect sync to solve cancer, climate change, aging, food insecurity, poverty, resource conflict, and every other problem that we could solve if only we had more time, wealth, or knowledge.
But it could also be an extremely bad thing.
We currently train AIs to behave the way we want by reinforcing them when they act in a desired way. This works well for the current generation of AIs, but there are serious worries that it won’t scale2, especially when the AIs become powerful enough to deceive their trainers.
Making sure future AIs are aligned with humanity is the most important problem of our time.
I strongly urge you to take action in this direction. The clearest step forward that I see if for the creation of an AI Safety Institute, exclusively focused on;
Developing methods to align increasingly powerful AI models.
Assessing models and alignment techniques developed elsewhere.
Working internationally to prevent the release of AI models which fail these assessments anywhere in the world.
This compares to CSIRO's National Artificial Intelligence Centre (NAIC), which has a broader mission3, and may face a conflict of interest with its role of encouraging the development and adoption of current AI systems.
There have been similar efforts in the US, UK, Canada, Japan and Korea. I consider the bill being proposed by California’s State Legislature to be the best in class so far.
Please do not let us fall into the trap where these existential risks from AI are conflated with the risks of technologies like social media. The attitude we need to have here is closer to regulating nuclear weapons, or to dual-use pathogen research, than to comparatively smaller problems like deep fakes, spam, or pornography.
I would be happy to talk to any of the points above in person or in future correspondence. I am working in Europe right now, so apologies if I’m unreachable by phone. I can always be reached by email.
Regards,
Hunter Jay
+61 422420037
____________
1 The heads of the leading labs lean to extremely short timelines:
“Sam Altman (OpenAI) believes AGI could be reached in four or five years.”
“Dario Amodei (Anthropic) believes human-level AI could arrive in two or three years, but “wouldn't say it's 100%. It could be 50-50”.”
“Jensen Huang (NVIDIA) expects that AIs will be “fairly competitive” with humans within five years.”
2 This is by Paul Christiano, the inventor of RLHF (the most commonly used modern alignment technique). He is arguing that although RLHF isn’t good enough to align smarter-than-human AIs, it was still probably net positive to develop. “I think it was valuable to use RLHF to fix the kind of basic alignment problems that are ubiquitous with pre-trained models. I think it has had a real impact facilitating work on more fundamental challenges, and helped move the community one step closer towards the kind of alignment solutions I expect to ultimately be successful.”
3 “The NAIC's mission is to amplify Australia's AI capability to create responsible and inclusive opportunities for every person, every business and the ecosystem.”