Embodied Intelligence Enters the Agentic Robotics Era? Tsinghua and Berkeley Release Unified Open-Source RUA Framework

Is Embodied Intelligence Entering the Agentic Robotics Era? Tsinghua and Berkeley Release Unified RUA Framework

Yesterday, Anthropic released Fable 5.2, bringing new advancements. Compared with previous models, its physical robot control capability has noticeably improved, even handling tasks it has never been specifically trained for before.

In the RoboDojo Sim simulation benchmark, Fable 5.2 scored 64.9%, far surpassing GPT-6 Astra (29.8%) in Anthropic’s internal tests with the same configuration.

As early as September, MIT researcher Phillip Isola proposed in “Robot-Use Agents” that in the future, robots connected to the Internet and with open sensor and actuator interfaces could become tools that general agents can invoke.

This also alignswith the System 1, System 2 division of labor in the embodied intelligence field.,where the general model handles high‑level reasoning and task planning, and the low‑level policy handles concrete action execution. GPT‑6 Astra,Fable 5.2 and other modelshave already preliminarily validated the feasibility of this approach.

However, to make it truly work, there are two key issues that need to be addressed.

The first is the model. Closed-source cutting‑edge models have already demonstrated impressive robot manipulation capabilities, but researchers cannot directly obtain the weights, making it even harder to continuously train them with robot interaction data.

The second is the system. Different robots differ significantly in hardware architecture, sensors, and control interfaces, and simulation and real environments also vary. If each robot integration requires redeveloping the tool system, the cost of cross-environment migration and large‑scale experiments will remain high.

Therefore, RUA needs a unified interaction interface that abstracts away the underlying differences of hardware and environments, allowing the model to obtain perception data, invoke control tools, and receive execution feedback in a consistent manner.

Recently, a research team from Tsinghua University including Ding Ning, Liu Zhiyuan, Xu Mengdi, Zhou Bowen, etc., together with Professor Pieter Abbeel’s team from the University of California, Berkeley and the Berkeley Artificial Intelligence Research (BAIR) Lab, jointly released and open‑sourced uni-robot-agent, aiming to build a unified, modular harness for Robot‑Use Agent (RUA) so that language models can control various robots and simulation environments via standardized tool interfaces.

• Paper link: https://arxiv.org/pdf/2611.01560

• Blog link: https://openrua.ai/uni-robot-agent

• Code link: https://github.com/Open-RUA/uni-robot-agent

This framework performed well in actual evaluations. On the RoboDojo benchmark, the open-weight model Qwen3.8-Flash-Next combined with this framework achieved a score of 31.39, surpassing the previously announced 28.97 score of GPT-6 Astra + RoboProbe by RoboDojo. Using the same framework, GPT-6 Astra further reached 51.90.

In the real-device evaluation of 19 tasks, Qwen without task-specific fine‑tuning scored 53.50, leading the fine‑tuned π₀.₅ and zero‑shot Dream-1 by 5.75 and 10 points respectively. GPT-6 Astra paired with this framework achieved 80.50.

Beyond the benchmark scores, this work also tests how far the model can operate a robot, uncovering failure reasons, capability gaps, and operational costs.

Furthermore, they aim to build a universal foundation that allows open-source models to interface with different robots, paving the way for subsequent training and local deployment.

Why a unified interface is the basis for cross‑environment generalization?

In the Coding Agent domain, the same set of tools can be used for different programming tasks; the model reads files, executes code, and checks results through similar interfaces, allowing it to reuse existing tool‑use experience across environments.

Robot operation involves even more hardware differences. Even when the task is simply grasping objects, the action interfaces of single‑arm and dual‑arm robots may differ, and the model needs to understand new parameters, coordinate systems, and control rules; existing tool‑calling experience cannot be transferred directly.

uni-robot-agent aims to bridge these differences through a unified tool interface.

For example, a Franka single‑arm robot needs to locate, grasp, and stack blocks; an ALOHA dual‑arm robot must coordinate two arms to hand over objects; a real YuMi dual‑arm robot must control the alignment and insertion of a plug into a socket. Although the hardware architecture, action space, and task objectives differ, the model can invoke unified perception, motion, skill, and task‑control interfaces, with low‑level adapters handling the control differences among robots.

In other words, the model can follow a consistent tool‑calling rule. When integrating a new environment, developers mainly need to implement the corresponding adapter, reducing changes to the upper‑level Agent logic.

Reuse of tools across different environments.

A unified interface also has a deeper significance.

If different environments use the same tool semantics, parameter units, and feedback format, the interaction trajectories generated by robots executing tasks can be more easily combined, and the experience accumulated by the model in single‑arm stacking, dual‑arm handover, and real‑world insertion can be consolidated into a single data system.

This design creates conditions for cross‑environment Agent training, giving researchers the opportunity to train a single model using interaction data from different robots and learn transferable tool‑use and manipulation capabilities.

How is uni‑robot‑agent designed?

The design of uni‑robot‑agent revolves around two goals. First, to provide a unified foundation for cross‑environment Agent training; second, to evaluate the existing robot manipulation capabilities of a general model as objectively as possible.

To this end, the research team divides the system into Agent, Orchestration, Tools, Services and Backends five layers.

The Agent layer handles interaction with the model, managing context and task decisions; the Orchestration layer schedules tool requests; the Tools layer provides a unified operation interface to the model; the Services layer manages runtime resources; the Backends layer connects to specific robots, simulators, and model implementations.

This modular architecture supports independent replacement of models, tools, and robot backends. Researchers can compare the performance of different models without rewriting the entire system, and accumulate interaction data with consistent structure.

Objective evaluation requires being as close as possible to real deployment conditions Simulation environments can directly provide the exact 3D positions of objects, CAD models, hidden scene states, and even judgments of task success, information that is usually not directly obtainable on real robots. If a model relies on these privileged cues to complete tasks, it becomes difficult to assess how much visual understanding, spatial reasoning, and manipulation ability it actually possesses.

Therefore, in the main simulation experiments, uni-robot-agent only provides the model with RGB images, camera calibration, robot body state, and task instructions, without offering depth ground truth, object CAD assets, hidden scene annotations, or task validator feedback, and external network access is prohibited. At the same time, the research team also restricts shortcuts specially crafted for the evaluation tasks, preventing the model from achieving high scores by using pre‑written task answers.

So, without this extra information, how can a general model perform complex robot manipulation?The key is to organize the information the model receives sensibly and ensure reliable tool execution.

Key design choices. Each column showcases two complementary designs, used respectively to ensure physical state consistency, action execution accuracy, and context management.

For example, when a robot grasps an object, the model must compute the grasp coordinates based on the image and the arm’s position; if the image is from a previous moment while the arm’s state has already updated, the calculation may be off.

uni-robot-agent binds camera images, calibration data, and robot state to the same timestamp, reducing positioning errors caused by state desynchronization. When executing actions, it deduplicates requests to avoid redundant operations, and monitors the arm’s motion state to detect stalls or execution anomalies.

After an action completes, the system feeds the execution result and new environmental observations back to the model, helping it determine whether the task succeeded and whether the next step needs adjustment.

To address the issue of long robot action durations, the framework also introduces asynchronous execution. The model can continue handling other tasks while some tools are running, and motion requests on the same robotic arm are queued sequentially to avoid command conflicts.

These designs incorporate engineering experience from the robotics field into Harness, reducing interference from information misalignment, action duplication, and missing feedback on model performance.

By organizing information sensibly and providing reliable control tools, uni-robot-agent can still enable large language models to demonstrate competitive robot manipulation capabilities without relying on privileged simulation data or task-specific shortcuts.

Experimental validation: what can open models achieve?

To evaluate the actual effect of the unified Harness, the research team first conducted simulation experiments on RoboDojo.

The benchmark includes 42 robot manipulation tasks, covering five dimensions: generalization, fine manipulation, long-horizon execution, memory, and open-vocabulary instructions. The main experiment performs 252 evaluations per group.

The impact of the unified Harness is very pronounced.

RoboDojo benchmark results. Avg. denotes the unweighted average of each metric across the five capability dimensions. †GAP (Hybrid) indicates that this approach incorporates π₀.₅.

Using the same Qwen3.8-Flash-Next model, RoboProbe scores only 1.90, RPent scores 7.75, and the GAP Hybrid that combines π₀.₅ reaches 21.01.

After switching to uni-robot-agent, Qwen’s score rises to 31.39, with a success rate of 25%. Compared with RoboProbe, the overall score improves by 29.49 points.

This set of experiments shows that even when the model weights remain unchanged, the design of the tool system can significantly affect robot performance.

The framework also supports other models. DeepSeek-V4.1-Flash scored 27.11, GPT-6 Astra reached 51.90, with a success rate of 46.67%.

Different models have different strengths. Qwen and DeepSeek perform relatively better on memory and open‑vocabulary tasks, whereas they are less strong in fine‑grained manipulation and long‑range execution; some specialized VLA and WAM models have an advantage on long‑range tasks.

Real‑world testing further evaluated the models’ capabilities in physical environments.

The research team used a dual‑arm robot to conduct 19 operational assessments, covering five dimensions: spatial understanding, object generalization, instruction following, compositional generalization, and unseen tasks. Specific tasks included rack‑inserting test tubes, dual‑arm bottle hand‑off, stacking blocks in color order, wiping a whiteboard, and solving the Tower of Hanoi.

Representative real robot tasks across five dimensions.

GPT-6 Astra scored 80.50 points, achieving a perfect 20 points in the combinatorial generalization dimension; Qwen scored 53.50 points, surpassing the task‑fine‑tuned π₀.₅ and zero‑shot Dream‑1.

Real-device test results. Each capability is scored out of 20, with a total maximum of 100.

However, this comparison involves differing input conditions. Qwen and Astra use RGB images and depth information provided by the RealSense D405 sensor, while π₀.₅ and Dream-1 use only RGB. Therefore, the performance differences cannot be entirely attributed to the models themselves.

What is more worth noting is exactly which external information the model relies on.

When both depth and camera calibration are provided, Qwen scores 35.89. Removing depth but keeping calibration yields a score of 31.39. Further removing calibration and retaining only RGB images and the robot’s own state drops the score to 9.13.

Astra achieved scores of 55.83, 55.04, and 31.29 under the respective settings.

Reduced robot performance under limited geometric information conditions. The figure records the process and final score of the model completing tasks in 42 scenarios with 771 tool calls when the skill library is enabled.

This indicatesthe model can leverage camera calibration, multi‑view observation, and geometric computation to accomplish partial spatial localization tasks without direct depth measurement. However, after calibration is removed, performance drops noticeably, and recovering reliable spatial relationships from RGB images alone remains challenging.

Whether a predefined skill library affects evaluation results is also a concern for the research team.

After disabling the skill library and skill authoring functions, Qwen’s overall score dropped only from 31.39 to 30.31, retaining 96.6% of its original score, and the success rate fell slightly from 25% to 24.58%.This shows that its performance does not primarily rely on pre‑packaged task skills; standardized perception and motion tools combined with the model’s own reasoning ability already support a substantial portion of the operations.

The impact of skill guidance on task performance.

In terms of cost, Qwen’s average API fee per task is about $1.09, DeepSeek is $0.94, and Astra reaches $21.29.

Execution speed also affects the efficiency of using robot agents. When the overall score first reaches 25 points, Astra has consumed about 7.4 million tokens in 42.6 minutes; Qwen consumed 23.5 million tokens in 95.5 minutes; DeepSeek consumed 49.9 million tokens in 99.2 minutes.

To reduce waiting time, the research team further tested asynchronous pipelines.

Across the five selected tasks, pipeline execution times were all shorter than serial execution, with reductions ranging from 4.8% to 73.7%. Notably, the conveyor‑belt grasping task time decreased from about 59.7 minutes to 15.7 minutes.

When will general models start to possess robot manipulation capabilities?

A unified harness allows different models to be tested under comparable interaction conditions, and also enables us to ask a more fundamental question:

What capabilities must a general-purpose model possess to accomplish robotic tasks?

The research team further compared 13 models from different families, generations, and scales, and found that robotic manipulation ability does not have a simple correlation with model size or release date.

Robot operation performance across different generations, model series, and parameter scales.

Taking the Qwen series as an example, the tested Qwen3.5, 3.6, and 3.7 models did not achieve any complete task successes. With Qwen3.8, the 27B model reached a success rate of 8.06%, and Flash-Next rose to 25%. Although the 27B model has a smaller parameter size, it can complete some tasks that older larger models could not.

The GPT series also shows generational changes. GPT-5.6 Sol can already handle some robot tasks, achieving a success rate of 19.17%; GPT-6.1 Sol improves to 37.50%, and GPT-6 Astra further reaches 46.67%.

These results indicate that robotic operation capabilities may have already emerged in general-purpose models, and their improvement could be related to training data distribution, capability composition, and model training methods.

By analyzing the failure trajectories of different models, the team identified four key capabilities.

The figure shows the correlation between each model’s overall score on RoboDojo and the error rates of the following four capability categories. The lower the error rates in these four dimensions, the better the model’s performance on robotic tasks.

First is visual understanding. The robot needs to recognize objects, determine orientation, and confirm the state after manipulation. Some earlier Qwen models mistakenly identified the digit 0 as 9, or judged an upside‑down cup as upright, causing subsequent actions to be based on incorrect observations.

In the inspected trajectories, this type of error occurred in older model versions at rates ranging from 12.2% to 23.8%. No such errors were found in the corresponding inspection samples for Qwen3.8-Flash-Next.

Next is spatial localization. The model must determine the three‑dimensional position of the target from images; when depth information is lacking, multi‑view observations can be used for triangulation, but matching the same physical location across views remains challenging.

The paper finds that even GPT-6.1 Sol still has about 31.5% of relevant triangulation calls failing the geometric consistency check.

The third category is geometric computation. Robots need to transform positions and poses between the camera, world coordinate system, and robotic arm coordinate system. Some older Qwen models have error rates in the relevant calculations ranging from 47.3% to 81.8%, Qwen3.8-Flash-Next drops to 8.2%, and DeepSeek-V4.1-Flash further reduces it to 1.5%.

Finally, there is embodied reasoning. The model also needs to determine whether the grasp pose is reasonable, whether the robotic arm can reach the target, and whether the action carries collision risk. Some models, even after finding the target position, repeatedly fail to grasp due to incorrect wrist posture.

The research team also correlated robot manipulation performance with general benchmarks such as ERQA, MMMU‑Pro, ScreenSpot‑Pro, and LiveCodeBench v6, finding that visual understanding, embodied reasoning, spatial localization, and coding abilities are each related to robot performance to varying degrees. Thereforethe ability to operate robots may represent a generalized combination of foundational capabilities across different domains.

Relationship between robot manipulation performance and general model capabilities.

Robot operation involves continuous visual judgment, spatial positioning, geometric computation, and physical decision‑making; an error in any link can cause the task to fail. To enhance RUA capabilities in the future, we need to focus on these interrelated foundational abilities simultaneously.

General framework,

What does uni-robot-agent support?

In addition to calling tools directly, uni-robot-agent also supports Code as Policy, Hybrid mode (VLM cooperating with VLA/WAM), and Clean Mode (retaining only basic tools). Different configurations can be tested within the same framework.

Code as Policy allows the model to control the robot by writing code; the model can perform calculations, invoke actions, and decide whether to continue based on execution feedback.

The research team used Qwen to compare code control and direct tool invocation, covering the five capability dimensions of RoboDojo. In 252 evaluations on RoboDojo, Qwen scored 25.60 points. Compared with the direct tool invocation results, code control achieved a higher score in the fine-grained operation dimension, reaching 17.40 points, but the overall performance still lags.

Score comparison of Qwen in Code as Policy mode.

The framework also supports Hybrid mode, enabling VLM and VLA to collaborate on tasks. VLM handles instruction understanding, scene analysis, and task organization, while VLA generates action segments based on observations, and the execution feedback is used for subsequent decisions.

The team demonstrated this collaboration in simulation scenarios on platforms such as the Panda robotic arm with the Allegro dexterous hand, the Yushu Go2 quadruped robot, the Yushu G1 humanoid robot, and the R1Pro wheeled robot, integrating action models like π₀.₅, GR00T, and NaVILA, covering dexterous manipulation, humanoid robot operation, and mobile navigation.

VLM and VLA collaborate in different environments via uni-robot-agent.

As model capabilities improve, some of the computation and operational logic currently encapsulated by tools can gradually be handed over to the model itself, and Harness is expected to become slimmer as a result. To explore this direction, uni-robot-agent offers Clean Mode, which retains only the four basic tools: move, hold, finish, and capture_scene, allowing the model to complete tasks by observing, calculating, and combining actions.

In a setup with camera calibration, Qwen and Astra under Clean Mode retain 90.4% and 90.5% of the scores of the full tool configuration, respectively, while reducing average token usage by 35.7% and 78.8%. Compared with the historical full configuration, the results show that after simplifying tools and context, the model can still maintain most of its task performance, providing preliminary evidence that Harness can be thinned and token efficiency improved.

Comparison of token consumption and scores for Qwen and Astra in clean mode.

The framework’s compatibility range has also expanded to more real robots.

The team has integrated ARX One, Songling Piper‑X and other physical devices. Control differences across hardware are handled by low‑level adapters, while the model still observes the environment, executes actions through a unified interface, and adjusts subsequent steps based on feedback. In addition, Zhiyuan G2 and Yushu G1 are also being integrated.

ARX One and Songling Piper‑X real‑device integration

The integration code and deployment instructions are also released, allowing developers to run uni-robot-agent on the corresponding hardware, swap models, and test their own tasks.

Conclusion

Robot intelligence is at a new crossroads.

Traditional dedicated‑policy approaches are tightly coupled to hardware, have high expansion costs, and face generalization and long‑range challenges. The uni-robot-agent approach lets general models control robots directly via a unified interface, resulting in a markedly different expansion logic.

uni-robot-agent is the first open‑source result released by the OpenRUA project. OpenRUA was launched by a group of researchers from top universities and cutting‑edge labs, with team backgrounds covering embodied intelligence and language models. Its goal is to build an open infrastructure for agents that can act in the physical world.

Team homepage: https://openrua.ai/

From this perspective, the significance of uni-robot-agent goes beyond providing a robot operation framework; it also demonstrates that, given well‑organized information and reliable tool feedback, open‑weight models can achieve competitive performance both in simulation and on real hardware.

However, this is only the first step. The next crucial part is whether we can continuously update the model using interaction data. uni-robot-agent has already set up the infrastructure, and the subsequent training and evolution are just beginning.

Reference links:

https://www.anthropic.com/research/project-pilot

https://www.anthropic.com/research/claude-plays-robotics

https://x.com/FredaDuan/status/2116727410972590380

© THE END

For reprints, please contact this public account for authorization

Submit or request coverage: liyazhou@jiqizhixin.com