An open-source computer-use agent with 397 billion parameters has beaten two closed systems roughly three times its size on the benchmark that measures whether an AI can actually drive a desktop. NorthCUA Pro, built on Qwen3.5, outperforms Qwen3.8-Max and Kimi-K3 on OSWorld-Verified and WebArena-Verified, according to a technical report posted to Hugging Face Papers and flagged by AI Weekly.
The result is a useful moment to ask what OSWorld actually measures, and whether a leaderboard win tells us these agents are ready for real work. Here’s what it means in plain terms.
What OSWorld Actually Tests
OSWorld is a benchmark that scores a model’s ability to use a desktop environment. Think of it like a driving test conducted entirely inside a computer: the agent gets a task, looks at the screen, moves the mouse, types, opens menus and tries to finish the job. A verifier then checks whether the end state is correct.
The “Verified” version used in the latest comparisons is the cleaned-up variant that most labs now report against. It matters because a score here is not a measure of how well a model talks about spreadsheets. It is a measure of whether the model can open one, change the right cell and save the file.
How We Got Here
Computer-use agents evolved from two directions that used to be separate. One camp had models read screenshots and click, which burns tokens fast: analyzing a 1920×1080 screen at every step means even a single spreadsheet edit can take many operations, and small coordinate errors or rendering delays can sink a task. The other camp ran shell commands and scripts, which are precise but clumsy at adjusting on-screen layouts or editing a drawing canvas that changes as you work.
Earlier attempts to combine the two relied on external routers or multiple specialized agents, which added handoffs, latency and lost context. Microsoft’s research team took a different route with Mage-VL-Cowork, a 4-billion-parameter vision-language model trained on both screen and terminal operation that decides for itself which channel to use. Its weights are on Hugging Face under the Apache-2.0 license, and it scores 66% on OSWorld-Verified.
NorthCUA’s authors describe a similar co-design philosophy. The report lays out an Experience Flywheel that supplies executable tasks, a Harness that “sustains long interactions, records decisions, and checks outcomes,” and a Staged Learning pipeline that moves from grounding mid-training through long-horizon supervised fine-tuning to joint desktop and web reinforcement learning. Training covered more than 4 million GUI interaction steps across dozens of applications, with trajectories collected in isolated E2B sandboxes and filtered by outcome.
The Numbers Across the Field
OpenAI’s GPT-5.4, released March 5, 2026, ships with built-in computer use and scored 75% on OSWorld-Verified. That compares with 47.3% for GPT-5.2 and 72.4% for the average human. Read that again: the model cleared the human baseline by 2.6 points, less than three years after its predecessor managed less than half of what a person does.
At the top of the current table sits Claude Opus 5.5, which scored 87.2% on the OSWorld offline subset. Google’s Gemini 4 Carbon, announced October 28, 2026, reached 81.5% on the same subset, trailing Opus by 5.7 points. Carbon leads 15 of 17 published benchmarks Google reported against Argon, GPT-6 Astra, Opus 5.5 and Fable 5.1, but OSWorld is one of the two it loses.
NorthCUA’s own claims are stated as comparative verdicts rather than numeric scores in the abstract. On OSWorld-Verified, “Pro outperforms Qwen3.8-Max and Kimi-K3; Mini surpasses Kimi-K3.” On WebArena-Verified, “Mini surpasses Qwen3.8-Max; Pro outperforms both.” The authors also ship a tradeoff variant, NorthCUA-Ultrafast, which pairs direct action selection with generation on the pre-RL Nano and completes matched successful WebArena-Verified tasks 2.3 times faster on average.
The Scoring Problem Nobody Talks About
A benchmark is only as good as the judge marking the exam, and that judge has been lenient. Researchers affiliated with Google and The University of Hong Kong built VeriCUA, a verifier that turns task instructions into checkable requirements and then looks for evidence in the app or file itself, using read-only online checks. When the evidence is inconclusive, it can abstain rather than guess.
On OSWorld, VeriCUA’s false-positive rate was 13.9%, against 25.9% for an auto-generated verifier and 27.9% for a vision-language model judge. A false positive means the verifier approved a task that was not actually completed. On AndroidWorld it recorded 90.5% accuracy and a 7.7% false-positive rate, versus 82.4% accuracy and 16.9% for the auto-generated alternative. On ScienceBoard the rates were 5.4%, 21.7% and 24.8% respectively.
The paper, submitted October 29, covered 116 AndroidWorld episodes, 361 OSWorld episodes and 169 ScienceBoard episodes, using each benchmark’s official evaluator as the reference label. VeriCUA issued no verdict in 22% of OSWorld episodes; among those where it did rule, accuracy was 88.7%. The evaluation counted abstentions as failures.
“A trajectory can appear successful even when edits remain unsaved or an operation fails without a visible warning,” Qiushi Sun and the other authors wrote. In other words, some share of every OSWorld score above may be measuring an agent that looked finished rather than one that was.
What the Rankings Say About Maturity
Three things stand out. First, the gap between open and closed is closing fast. A 397-billion-parameter open model beating trillion-parameter closed systems on a desktop benchmark would have been a surprise a year ago; now it is a technical report on Hugging Face Papers. Second, size is no longer the whole story: Microsoft’s 4B model clears 66%, and NorthCUA’s 35B Mini beats one trillion-parameter rival on each benchmark.
Third, and most important for anyone hoping to hand an agent their expense reports, the field is now mature enough to have a measurement crisis. When a quarter of approvals under one judge are wrong, a two-point leaderboard gap is noise. VeriCUA’s abstention rates, 18% to 30% depending on the benchmark, are a blunt reminder that many agent runs land in a gray zone where nobody can say for certain whether the job got done.
The practical upshot for readers testing these tools at home: run your own verification. Check that the file saved, the email sent, the row updated. The benchmark is catching up to that habit. VeriCUA’s first online evaluation used 39,200 tokens over 97 seconds per pass, against 60 seconds and 16,200 tokens for the auto-generated verifier, and the paper frames verification as a way to evaluate agents, curate training data and assign rewards rather than a product.
NorthCUA’s release landed into a dense stretch of open-source agent work. AI Weekly’s agents tracker counts 527 stories in the last 90 days, with the Tsinghua and Berkeley Uni-Robot-Agent harness crossing the wire the same day. The next number to watch is not another leaderboard entry but a verified one: how many of those 87.2% and 75% scores survive a stricter judge.
Sources
- NorthCUA Pro 397B Tops Qwen3.8-Max, Kimi-K3 on OSWorld-Verified | AI Weekly
- GPT-5.4
- Google and Hong Kong Researchers Find Fewer False Approvals in AI Agent Tests | TokenPost
- Microsoft Releases Mage-VL-Cowork, a 4B-Parameter Computer-Use Model That Scores 66% on OSWorld-Verified
- Gemini 4 Carbon launch: Google's access and tests | NeoTeo
- Google DeepMind Introduces Gemini 4 Carbon, Its Top Model for Cyber Defense and Coding
Photo: Anastassia Anufrieva / Unsplash (Unsplash License)
