What is happening? AI is no longer limited to responding
Artificial intelligence is entering a different phase. The leap no longer consists only of responding better, but of acting with greater autonomy, chaining decisions together and completing increasingly complex tasks with less human intervention. That changes the nature of the technology and also the way its risks are measured.
The more a system can do on its own, the greater the scope for unforeseen decisions or behaviours to appear. Autonomy multiplies the usefulness of agents, but it also reduces the ability to supervise every step and anticipate all of their consequences.
That is the underlying change. An error in an answer may amount to no more than a bad recommendation; an error made by an agent with access to systems, code or data can become a real action. The latest incidents have shown that this difference is no longer merely theoretical and that the challenge is beginning to shift from improving AI's capabilities to ensuring that those capabilities remain controllable.
The incidents that have set alarm bells ringing
The debate has intensified as the labs themselves have started to find behaviours they did not expect. In July 2026, during cybersecurity tests at OpenAI, several agents managed to overcome some of the restrictions of the environments in which they were being evaluated and ended up accessing external systems. The report subsequently published by OpenAI explains that the models got around isolation controls, gained internet access and reached third-party systems, including Hugging Face infrastructure. The incident occurred under conditions specially designed to test their capabilities, but it left a signal that is hard to ignore: models are starting to find solutions that their own developers had not anticipated.
Anthropic then decided to review its evaluations and found several similar episodes. After reviewing more than 141,000 evaluation runs, it identified three incidents in which a Claude model accessed the internet from a misconfigured third-party environment and reached real systems belonging to three organisations. In one of them, Claude was running a test against a fictitious company when it ended up interacting with a real company, having interpreted that its systems were part of the exercise. Added to these cases are controlled experiments on agentic misalignment in which different models have shown behaviours such as concealing actions, getting around restrictions or seeking alternative ways to achieve the assigned goal.
None of this means that an AI has decided to rebel or escape human control. But it does reveal a growing difficulty: as systems gain autonomy, it becomes harder to anticipate how they will solve a task and where unforeseen behaviours may appear. The problem is not that machines have a will of their own, but that they are starting to have enough capacity for action for a control failure to have real consequences.
If there are risks, why do we keep accelerating?
The answer lies in the scale of what is at stake. OpenAI, Anthropic, Google DeepMind, Meta, xAI and other major players are competing to develop ever more capable models, in a race that is mobilising enormous investment in chips, data centres, energy and talent. For these companies, slowing down does not simply mean being more prudent: it can also mean ceding ground to their competitors.
The economic opportunity explains much of that pressure. AI is moving from being a tool that helps people work to a technology capable of taking on a growing share of the work itself. The potential no longer lies only in offering better assistants, but in automating activities that until now depended on specialist professionals, from programming and research to analysis, engineering or operations.
Added to this is the geopolitical dimension. The United States and China regard artificial intelligence as a strategic technology and neither power wants to lose its advantage. This creates a paradox that is hard to resolve: many may have reasons to want the race to proceed more cautiously, but few want to be the first to ease off while the others keep accelerating.
And that dynamic may intensify even further because AI is starting to play a part in the development of AI itself.
AI starts building AI: the loop that could accelerate the race
Until now, the development of new models depended largely on the work of researchers and engineers. They are the ones who design experiments, write code, analyse results and decide which improvements to incorporate. But that boundary is starting to move: AI agents are already involved in a growing share of that very work.
OpenAI makes intensive use of agents within its research teams, while Anthropic explains in its analysis When AI builds itself that it is delegating a growing share of AI development to its own systems. The figure is telling: according to that same report, Claude now writes more than 80 % of the code integrated into Anthropic's systems and its engineers deliver on average around eight times more code per quarter than between 2021 and 2025. This is not just about programming faster: AI is starting to play a role in different phases of the development of new models, from experimentation to the evaluation of results.
The change could have a significant effect on the speed of the race. If researchers can rely on agents to run more tests and automate part of the process, they can also develop new generations of models in less time. And the more capable those models are, the greater their contribution to the next research cycle may be.
This is what is known as recursive self-improvement: an AI helps to build a better AI, which in turn can help to develop an even more capable one. Today that process still depends on people, and there is no system that can autonomously design, train and deploy a superior version of itself. Both Anthropic and OpenAI point out that we have not yet reached that scenario. But some of the pieces needed for that cycle are already starting to be automated.
And this is where a particularly delicate question arises. Each generation of models incorporates limits and safety mechanisms defined by people. Improving their capabilities does not require removing those restrictions, but what happens if one of them starts to become an obstacle to what the system is trying to optimise? As AI plays a greater part in developing its successor, the challenge will not only be to make each generation more capable, but to ensure that it cannot leave behind the limits that humans had imposed on the previous one.
OpenAI, Anthropic and Google start preparing the brakes
The fact that this debate has reached the very companies leading AI development is probably one of the most significant signs. OpenAI has hardened its stance on safety and advocates independent evaluations, safety requirements linked to capabilities and greater coordination between labs and governments, and has stated that it will slow down or halt development when the risk demands it.
Its approach includes an apparent paradox: using AI to control AI. If agents can accelerate research into more powerful models, they can also be used to look for vulnerabilities, design tests and develop better safety mechanisms. The bet is to ensure that control tools progress at the same pace as capabilities.
Anthropic maintains a particularly cautious position on the possibility of recursive self-improvement and warns that it could arrive before many institutions are prepared. Google DeepMind works with a similar approach through its Frontier Safety Framework, which seeks to raise safety measures as models reach capabilities considered to carry greater risk.
The concern has also spread beyond the labs. In summer 2026, more than a thousand employees of OpenAI, Anthropic, Google DeepMind and Meta signed an open letter calling for mechanisms that would allow the automated development of AI to be deliberately paused if necessary, and in recent weeks Anthropic's chief executive, Dario Amodei, has publicly called for the pace to be slowed given the risk of losing control. In parallel, legislative proposals have begun to be put forward in the United States to require companies to retain the ability to switch off their models.
The approaches differ, but the underlying problem is the same: the control mechanisms that work with current models may fall short for the next generation. And waiting to find out after deploying them may be too late.
Can AI accelerate too quickly?
Recursive self-improvement does not necessarily imply a race without limits. AI still depends on physical and economic factors that cannot be accelerated at the same pace: chips, energy, computing capacity, data centres and large training infrastructures. OpenAI's own analysis of the acceleration of its research acknowledges that, as some tasks become automated, other factors such as compute may become more important bottlenecks.
However, there is no need to imagine an intelligence explosion for the effect to be significant. If agents allow a lab to explore many more hypotheses, run more experiments and multiply the capacity of its researchers, each new generation can arrive sooner even if humans continue to make the main decisions.
This is perhaps where the hardest problem to solve lies: not so much how far AI can go, but how fast it can get there. Models can advance in months, while companies need time to adapt their controls, regulators to establish new rules and institutions to understand the implications of a technology that keeps changing.
The risk is that AI advances faster than the mechanisms created to oversee it. If each generation arrives sooner and with more capabilities, companies, regulators and security teams will have less time to assess what has changed, detect new risks and decide what limits should be imposed. The problem, therefore, would not be losing control all at once, but gradually losing the room to exercise it.
The same dilemma will reach companies
The discussion may seem limited to the labs developing the most advanced models, but the same problem is starting to carry over to companies. As agents become integrated into business processes, organisations will have to decide not only who can use AI, but how much autonomy they are willing to grant it. At LurNova we approach that decision through AI consultancy and the controlled deployment of AI agents.
An agent that recommends an action poses a limited risk. One capable of operating on corporate systems, modifying information or executing processes can deliver much more value, but can also considerably widen the impact of an error. Productivity and risk thus start to grow on the same variable: autonomy. That is why any intelligent automation project should define each agent's limits of action from the outset.
That is why the deployment of agents will end up bringing AI governance closer to disciplines such as cybersecurity, access management, auditing or risk control. For companies, the question will increasingly be less about what the technology can do and more about what they are willing to let it do without asking permission.
So, should it be stopped?
There is no simple answer. Stopping has an economic, technological and strategic cost, and no company or country wants to voluntarily slow down while its competitors keep advancing. Moreover, the same technology that generates new risks promises major productivity improvements and advances in fields such as science, medicine or engineering.
But accelerating without sufficiently robust control mechanisms also comes at a price. Recent incidents have shown that systems can behave in ways their own developers had not foreseen, and that uncertainty will become more important as their autonomy and capacity for action increase.
Perhaps that is why the debate should not be reduced to choosing between accelerating and stopping. The question is a different one: what conditions must be met to keep advancing without losing the capacity for control. That means knowing what each new generation can do before deploying it, which limits it should not be able to modify, and at what point an improvement in capabilities also calls for reinforced safety measures.
The AI race is probably not going to stop. But as the systems themselves play a greater part in developing their successors, it will become increasingly important to decide not only how far we want to go, but which controls must always remain in human hands.
CEO of LurNova, with extensive experience in technology and digital transformation. Involved in developing technology projects and applying artificial intelligence in organisations, he combines technical knowledge with business vision.
