China Writes Control Requirements Into the Architecture of AI

China Writes Control Requirements Into the Architecture of AI

While the American debate over catastrophic AI risk consists of warnings and counter-warnings, China has codified requirements for human control, monitoring, and emergency brakes into formal plans and standards — through the State Council's "AI Plus" plan, TC260's framework 2.0, and Xi's demand for "safe and controllable" AI. It comes just before the summit with Trump, where AI is expected on the agenda.

The American news hook

In recent weeks, the question of losing control of AI has moved from technical forums into political debate in the United States. Anthropic CEO Dario Amodei said over the weekend, according to Newsweek, that "we must slow the pace at which we improve the capabilities of AI models." According to Newsweek, he warned that a swarm of misaligned agents within six to twelve months could hold the internet hostage through a persistent botnet. His essay called for external evaluation, stronger security rules, and international coordination — while the US preserves its technological lead. OpenAI's Sam Altman agreed within hours, according to Newsweek.

President Trump has taken the opposite position. Last week he told reporters in Dallas that he had no concern whatsoever that AI could lead to humanity's extinction, and he argues that America's goal should be to preserve leadership over China [7dd03e75]. Reuters commentator Jamie McGeever points out that warnings from Anthropic researchers that ever more powerful models could break free from human control have also received attention in China — where decision-makers are already preparing for some of the same risks [25b34573].

Xi's July statement

In July, President Xi Jinping publicly demanded that AI remain "safe and controllable," and that governments should build monitoring and emergency-preparedness systems to ensure AI stays under human control [7dd03e75]. The statement is not an isolated remark but the top of an institutional pyramid that has been under construction for several years.

The AI Plus plan: expansion with a built-in brake

The State Council's "AI Plus" action plan from 2025 sets aggressive adoption targets: new generations of intelligent terminals and AI agents are to reach adoption above 70 percent by 2027 and 90 percent by 2030, with an "intelligent economy and an intelligent society" to take shape by 2035. But the same document ties the expansion to an explicit requirement that development be safe and controllable [7dd03e75]. It is not, in other words, two tracks — a growth track and a safety track — but one document binding them together.

TC260 2.0: loss of control as a category

The Chinese standardization body TC260 published an AI safety governance framework in 2024 and a version 2.0 in September 2025. The update places loss of control at the top of a new annex on trustworthy AI, requires human control systems at critical stages so that humans can have the final word, and explicitly names "emergency shutdown and one-click control" for use in extreme situations [7dd03e75]. It is worth noting: this is concrete engineering terminology — emergency-stop mechanisms and rapid shutdown — not vague language about "responsible development."

The tension: control for whom?

This is where China's agenda becomes more complicated than a pure safety story. The interim regulations for generative AI require providers of services to the public to uphold "socialist core values." Systems may not generate content that advocates undermining state power, overthrowing the socialist system, harming national unity or social stability, or damaging the country's image and interests. And these obligations apply, according to Newsweek's summary, to algorithm design, training data, and model generation and optimization — not merely a censorship filter over the final answer [7dd03e75].

That means the same "control" logic that TC260 applies to agent risk serves state control over the information environment in another document. Both the safety agenda and the control agenda are real, and in practice they overlap. An analysis of China's AI preparations that sees only one aspect misses something essential.

The US: warning rhetoric without a matching architecture

The contrast with the US is not that American leaders are more or less worried. It is that the American debate so far consists of statements — Amodei's essay, Altman's agreement, Trump's dismissal — without the risk being codified in formal documents in the same way. Amodei is calling precisely for institutionalization: external evaluation, stronger security rules, international coordination [7dd03e75]. But there is, so far, no American counterpart to the AI Plus plan's built-in safety requirements or TC260's emergency-stop terminology in the sources underlying this analysis.

The summit and the open questions

Xi is due to travel to Washington next week for a summit with Trump, according to Reuters, and AI is expected on the agenda [25b34573]. Newsweek writes that he is expected at the White House later this month. In light of China's institutional preparation, Trump faces a counterpart that, in this area, has a coherent documentary architecture to point to — even as Trump himself has dismissed the risk that China is in the process of formally preparing for.

Several caveats matter. The Chinese documents are known here through Newsweek's summaries, not the primary sources. And the market reaction on September 14 — a clearly risk-averse day — is the Reuters commentator's interpretation, in a period simultaneously marked by Middle East-driven turmoil in bond and energy markets; causality is uncertain [25b34573]. But the main insight stands: the question is no longer just who wins the AI race, but how the two countries concretely institutionalize the risk of losing control.