App diversity lifted OpenMobile-2's success rate from 60.7 to 67.5 percent

App diversity lifted OpenMobile-2's success rate from 60.7 to 67.5 percent

A Chinese research team has opened the entire recipe: two Qwen-based agent models, a training environment with over 110 simulated apps, and the datasets and instructions to reproduce everything. But the open models' progress rests on training data generated by Google's closed Gemini 3.1 Pro.

A research team from Nanjing University, SenseTime, and several partner universities has released OpenMobile-2, an open-source project that trains AI agents for smartphones. The release, covered by The Decoder on October 25, 2026, consists of two agent models built on Qwen3.5 with 9 billion parameters and Qwen3.6 with 27 billion parameters, together with the complete training environment, the data, and instructions to reproduce everything.

What makes the release worth a closer look is not primarily the benchmark numbers, but the environment behind them: a simulated app store with over 110 apps, and a finding that variety in apps — not the amount of data — was what lifted the agents' performance. At the same time, one detail about the data complicates the simple picture of open models challenging proprietary systems.

Why they built their own app store

The training ground for OpenMobile-2 combines existing simulations with apps running in an Android emulator, for a total of more than 110 applications. Among them are 35 rebuilt apps popular primarily in China — Taobao, JD, DiDi, and Meituan — plus globally widespread services like Slack and Google Drive.

The choice to rebuild the apps rather than train on real ones is not accidental. According to The Decoder's rendering of the researchers' rationale, training on real apps is painful because logins, real purchases, and the absence of a way to reset state make the process expensive and hard to repeat. An agent practicing in your Taobao app can order items you don't want; an agent failing in a logged-in session can destroy the very state it was supposed to learn to navigate. Simulated copies solve this: they can be reset, run at scale, and run in parallel with no real-world consequences.

From this setup, the team generated roughly 11,600 example workflows and over 2,000 practice tasks with automated success checks. The automated checks are a key component: they make it possible to evaluate thousands of agent attempts without human inspection of each individual click.

The main finding: variety beat volume

The most interesting result from OpenMobile-2 is not about how large the models are or how much data they received, but how the data was composed. With the same amount of training data, increasing the number of apps raised the success rate on unseen applications from 60.7 to 67.5 percent.

The reader should note what this comparison actually isolates. The models saw just as much training data, but the data covered a broader range of apps. When performance on apps the model has never seen improves by nearly seven percentage points, it suggests the agent learns something general about how mobile interfaces are structured — buttons, lists, forms, navigation flows — that transfers across apps, rather than memorizing solutions for individual apps.

The implication for how agent training should be resourced is concrete: if the finding holds, it is better to spend effort building the environment broadly — more apps, more interface patterns — than to generate yet more examples in a few apps. It is a claim about training economics that can be tested, and that other research groups can now verify, since the entire environment and instructions are open.

The numbers, and what they show

According to The Decoder's rendering of the benchmark results, OpenMobile-2 delivered clear gains on the standard tests:

• On AndroidWorld, the larger model went from 67.1 to 79.9 points, while the smaller went from 54.3 to 74. The predecessor OpenMobile-1 scored 64.7 on the same test.
• On MobileWorld, the larger model improved from 35.9 to 50.4 points.
• On real Android devices, the larger model scored 59.6 points on SPA-Bench, the highest result among open models.

The numbers tell two stories. The first is that the methodology works: both models improved clearly, and both beat the predecessor's 64.7 points on AndroidWorld. The second is more nuanced: the SPA-Bench result on real devices is described as the highest among open models — not the highest overall. Open models have thus taken the lead in their own class; the distance down to closed commercial systems cannot be read from the reported numbers alone.

It is worth clarifying that all of these figures come via The Decoder's coverage. This article has not verified them against the researchers' own paper or codebase, so they should be read as reported results, not first-hand documented ones.

The closed teacher behind the open models

Here comes the caveat that makes the "open beat closed" narrative more complicated than headlines suggest. Google Gemini 3.1 Pro served as the teacher model that produced the examples — the roughly 11,600 workflows the agents were trained on. As The Decoder puts it, this means OpenMobile-2 effectively learns from a closed system.

This is a well-known pattern in modern AI research: a strong, closed model is used to generate training data that is distilled into a smaller, open model. Open-source weights, environment, and instructions are real and valuable — anyone can download, run, study, and build on OpenMobile-2. But the capacity to generate the knowledge in the data itself comes from a system that no one outside Google can inspect.

That has three consequences. First, the project is not independent of closed frontier models; it depends on them as a data source. Second, any biases or limitations in the teacher model will be inherited, invisible in the open weights. Third, it sets a structural limit: the methodology shows how to build good open agents, but it does not show that open research can generate comparable teacher data without access to a frontier model — whether one's own or a rented one.

What the evidence does not say

Several questions remain open after what is actually documented in the coverage.

There is no verified direct comparison with proprietary agent models in the available material. The Decoder's headline sets up the idea that open agents can measure themselves against closed models, but the cited figures show results among open models, not a top ranking across the entire field. It may well be true that the gap has closed, but that requires its own comparison numbers not present in the material this article is based on.

The researchers' own paper or codebase has not been verified here; all numbers come from The Decoder's rendering. Nor do the research's publication date or peer-review status appear in the coverage, dated October 25, 2026.

To that come the questions the methodology raises: How robust is the app-diversity finding across model families other than Qwen? How much of the performance is due to distillation from Gemini 3.1 Pro itself, independent of the environment? And how do the agents behave in real apps with all the pitfalls the simulation was designed to avoid — logins, payments, and state that cannot be reset?

Why this matters

Regardless of these open questions, the release is significant on three counts.

It gives the research community a complete, reproducible training environment for phone agents, with over 110 apps and automated success checks — infrastructure that has previously been a bottleneck because training on real apps is expensive and irreversible. It delivers a measurable finding that app diversity yields more than data volume, with numbers (60.7 to 67.5 percent on unseen apps) that others can now test. And it shows that open models can reach the top of their class on benchmarks like SPA-Bench, with 59.6 points on real Android devices.

But the most lasting lesson may be the dual one: the environment and the weights are open, while the knowledge in the data is distilled from something closed. For anyone who wants to build on OpenMobile-2, that is good news — the code and data are there. For the larger story of open versus closed AI, it is a reminder that "open" rarely means independent.