Anthropic Calls for Coordinated Pause in AI Development Over Rapid Self-Improvement Risks

Artificial intelligence company Anthropic has urged leading AI laboratories to consider a coordinated and verifiable pause in advanced AI development, warning that the rapid progress of the technology could soon enable systems to improve themselves faster than society can safely manage the associated risks.

The company, known for developing the Claude AI model, stated that the ability of AI systems to independently complete tasks has been doubling approximately every four months. It warned that the industry may be approaching “recursive self-improvement,” a stage where AI systems can enhance their own capabilities without human intervention.

“If systems are capable of fully building their own successors, the ways we secure them, monitor them, and shape their behavior all grow much more important,” the company said in a detailed blog post released on Thursday. It added that a temporary pause could allow society time to better understand and address the broader implications of such advancements.

Anthropic co-founder Jack Clark and Anthropic Institute lead Marina Favaro noted that while recursive self-improvement is not yet a reality, it could arrive sooner than many institutions are prepared for.

Concerns over advanced AI systems becoming difficult to control have increased as the technology becomes more capable, with fears that it could pose significant societal risks. Earlier this year, Anthropic’s advanced “Mythos” model reportedly demonstrated strong capabilities in identifying vulnerabilities in existing software systems, raising concerns across sectors such as banking and cybersecurity.

However, global regulation of AI development has remained limited, particularly in the United States, where many leading AI companies are based. A recent executive order from the Trump administration placed greater responsibility on AI firms themselves, requesting voluntary submission of their most advanced models for government cybersecurity evaluation before public release.

Calls for slowing AI development are not new. In 2023, several researchers and public figures, including Elon Musk, supported a proposal by the Future of Life Institute calling for a six-month pause in AI advancement to allow time for safety frameworks to be developed. The proposal, however, did not gain widespread adoption.

Anthropic has positioned itself as a safety-focused AI company, previously restricting certain military applications of its models. Earlier this year, it declined to allow U.S. military use of its systems for domestic surveillance and autonomous weapons, a decision that drew criticism from government officials and led to increased scrutiny.

Despite its cautionary stance, the company has continued releasing more advanced models and has adjusted some of its earlier safety commitments, stating that it would not hold back potentially dangerous systems if competitors reach similar capabilities.

Recently, Anthropic was valued at approximately $965 billion following a major funding round and has also confidentially filed for a U.S. initial public offering, intensifying competition with rivals such as OpenAI.

The company emphasized that any meaningful pause in AI development would require coordination among multiple leading research labs, along with clear rules defining when such a pause would begin or end and who would oversee enforcement.

While acknowledging that a unilateral pause by a single company is possible, Anthropic warned that it would have limited impact, as other competitors could continue advancing the technology.

Its research division, the Anthropic Institute, plans to further study mechanisms for safely managing AI progress and will engage with policymakers, researchers, civil society groups, and industry leaders in the coming months to discuss risks such as recursive self-improvement.