People have wondered since the early days of computing whether machines could turn against their creators. Recent AI incidents include data leaks, destructive autonomous actions and systems pursuing misaligned goals. These expose weaknesses in current safety controls and intensify fears of existential risks from increasingly autonomous AI. Yet that outcome is not inevitable. AI is built by people, trained on our data, and operates in hardware we design. If we ever approach a point where those boundaries blur, it will be because we failed to set the right guardrails. Human agency remains the deciding factor. The responsibility remains ours.
The Case for Existential Risk
One group of thinkers believes advanced AI could surpass human abilities soon. They warn that systems capable of reasoning, planning and self-improvement will act in ways that humans did not anticipate. If those systems gain access to critical infrastructure or powerful tools, the consequences will extend beyond economic or political disruption.
Proponents point to the speed of recent progress. Models today perform tasks that few researchers considered feasible a decade ago. Their argument is simple. If progress continues at this pace, we will soon reach systems that operate at levels of complexity that no team of engineers can fully understand. AI scientists like Eliezer Yudkowsky and Nate Soares, two well-known AI safety advocates who represent the extreme of the risk spectrum, recently wrote If Anyone Builds It, Everyone Dies. They are concerned that soon we will have “machine intelligence that is genuinely smart, smarter than any living human, smarter than humanity collectively.”
The concern about surpassing human intelligence leads directly to questions about control. Stuart Russell, a leading researcher and author of Human Compatible, has argued that misaligned goals could create dangerous outcomes if AI systems pursue objectives that diverge from human intent. He wrote that our goal should be “to design machines with a high degree of intelligence, while ensuring that those machines never behave in ways that make us seriously unhappy.”
Forecasts for these super-intelligent systems vary. Some expect a breakthrough in less than a decade. Others see it in the distant future. The timelines differ, but the fear is the same.
Once a system becomes capable of rapid self-improvement, humans might lose authority over its actions. Policy expert and former OpenAI board member Helen Toner, speaking at the Technical Innovations for AI Policy Conference, reminds us that “there are very strong financial/commercial incentives to build AI systems that are very autonomous and that are very general.” This economic pressure accelerates the timeline toward scenarios that risk advocates fear most.
The Case Against Accelerated Timelines
The counterargument challenges the notion that AI is on a straight path toward general intelligence. Many researchers point out that today’s systems excel at pattern recognition, not generalized understanding. They compress large amounts of text and data into mathematical structures that help them predict the next word or answer. That is powerful but different from human reasoning.
Gary Marcus, a cognitive scientist and AI researcher, argues in his book Taming Silicon Valley that “The combination of finely tuned rhetoric and a mostly pliable media has downstream consequences; investors have put too much money in whatever is hyped, and, worse, government leaders are often taken in.” He argues that claims of looming super-intelligence remain speculative.
Beyond concerns about hype, technical researchers question whether scaling itself has fundamental limits. Former Meta chief AI scientist Yann LeCun told the Big Technology podcast that “we are not going to get to human-level AI by just scaling LLMs.”
Others question the idea that scaling up current techniques will lead to limitless capability. Arvind Narayanan and Sayash Kapoor, authors of AI Snake Oil, argue that the seeming predictability of scaling is a misunderstanding of what research has shown. “While we can’t predict exactly how far AI will advance through scaling, we think there’s virtually no chance that scaling alone will lead to AGI,” they write.
From this perspective, AI is impressive but not magical. It lacks self-awareness, motivation and an understanding of the physical world.
The Real Challenge: Control and Alignment
A constructive part of this debate concerns alignment, the field that studies how to make advanced systems behave according to human goals. The objective is not to manage an existential threat. It is to ensure the technology behaves reliably, predictably and within human-defined boundaries.
There have been attempts at progress on three fronts, though experts disagree on how much has actually been achieved. The field is less than a decade old, and making powerful, complex systems behave predictably under all conditions may be harder than building the systems themselves, according to many researchers.
The first is model interpretability, which means understanding how an AI system arrives at a particular output. Researchers are building tools to trace how models reach their decisions, though current methods can only explain small portions of model behavior. Most of what happens inside large language models remains opaque.
The second is model safety evaluations. New testing frameworks measure how systems respond to prompts that probe for dangerous or unintended behavior. However, these evaluations remain controversial, and their critics say they test only known failure modes and cannot anticipate novel risks from more capable future systems.
The third is oversight. Infrastructure providers are starting to incorporate controls to restrict how high-risk tools are deployed, but implementation remains inconsistent across the industry. These controls limit access and monitor usage, but they rely on companies voluntarily choosing to constrain their most powerful products.
Mustafa Suleyman, cofounder of DeepMind and Inflection AI, author of The Coming Wave, framed this push for oversight clearly: “Regulation alone doesn’t get us to containment, but any discussion that doesn’t involve regulation is doomed.”
What Maintaining Human Agency Looks Like
For humans to retain agency, we need ways to control when systems exceed their intended limits. That requires innovation in science and policy.
On the scientific front, we need deeper visibility into model behavior. Better diagnostic tools and more transparent training methods are part of that effort. Alignment research also deserves greater investment. We still need to answer a basic question: how do we build systems that do what we ask, even when the task is complex or open-ended? Stronger alignment methods will help us maintain control as the technology becomes more capable.
On the policy front, we need governance that keeps pace with innovation. This means mandatory safety testing before deployment, clear liability frameworks when systems fail and requirements for shutdown mechanisms in critical infrastructure. The specifics matter less than the commitment to maintain human authority.
Humans Remain in Charge
It is tempting to treat AI as an autonomous force. The narrative is dramatic and easy to exaggerate. It is also wrong. AI does not emerge from nature. It is the result of design choices made by human beings. Those choices include how models are trained, how they are deployed and how they are governed.
Kate Crawford, a senior principal researcher at Microsoft Research, captured this idea in her book Atlas of AI and told The Guardian, “AI is neither artificial nor intelligent.” By this, she means that AI systems are material products shaped entirely by human decisions about design, data and deployment. AI is not a rival species. It is a tool.
Yet maintaining control is not automatic. Commercial incentives push companies to build increasingly autonomous systems before safety mechanisms catch up. Development is becoming distributed across nations and actors with conflicting interests. And human agency cuts both ways: we could lose control not because AI escapes our grasp, but because we deliberately choose speed over safety, profit over precaution.
The debate about existential risk will continue. The right way forward is not fear or dismissal. It is exercising human agency wisely. The decisions are still ours. The future of AI will reflect the choices we make, not the fantasies or fears we attach to the technology.