AI startup Anthropic has warned that AI is rapidly approaching the point of 'recursive self-improvement', where it can design its own successor. The firm highlights the risk of losing control and calls for a slowdown in development.
AI startup Anthropic has issued a warning that the fast-paced evolution of the Artificial Intelligence technology is reaching a point where it could design and develop its own successor. The startup, in a report, said that the company is "delegating a growing share of AI development to AI systems themselves, which is speeding up our work".
"Taken far enough, and given enough compute, that trend points to an AI system capable of fully autonomously designing and developing its own successor," Anthropic said. Calling it recursive self-improvement, the San Francisco-headquartered AI firm said that the world is not there yet, but it could come sooner than most institutions are prepared for.
AI Boom and Company Valuation
The Claude-maker recently raised $65 billion in a funding round valuing the company at close to a trillion dollars. It has confidentially filed for an IPO amid a boom in AI shares as investors continue to pour money in AI stocks, remaining bullish on the prospects.
Accelerating Progress and Emerging Risks
Just to show the scale of how fast AI is developing itself, the company said that Anthropic engineers, on an average, are shipping 8x as much code per quarter as they did from 2021-2025.
The company said that while a self-improving AI is good for the overall development, a full recursive self-improvement risks humans losing control of AI systems. "If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behaviour all grow much more important," Anthropic warned.
The extent to which the AI models can finish a task on their own is doubling roughly every four months. "In 2027, AI systems could be capable of tasks that take a person weeks," the company said.
A Call for a Development Slowdown
Anthropic urged for slowing down of the AI development process to give more time to come up with technology to deal with its immense implications. "A meaningful slowdown or pause would require multiple well-resourced labs at or near the frontier, in multiple countries, agreeing to stop under the same conditions. It would also require that each can verify that the others have actually stopped."
(Except for the headline, this story has not been edited by Asianet Newsable English staff and is published from a syndicated feed.)