Can World Models Be Put to the Test? WorldOracle Presents a Real-World Exam

Can World Models Be Put to the Test? WorldOracle Presents a Real-World Exam

Machine Heart Release

The robot clearly missed, but the world model predicts that it successfully grasped the fruit.

This is a case presented by WorldOracle: given the initial screen of a real operation and a sequence of actions, the model generated a result that looks plausible but differs from the actual execution. The success shown on the screen did not occur in reality.

Is a world model actually predicting the future, or is it “imagining” the most common successful outcome?Recently, researchers from Princeton University, the University of Texas at Austin, the University of Toronto, NVIDIA, and Zhiyuan jointly proposed WorldOracle, which uses the real execution corresponding to the same initial scene and action sequence as a reference to evaluate the predictive capability of world models.

We studied seven representative models including DreamDojo, Cosmos 3, IRASim, GE‑Sim 2.0 and other models, conducting evaluations and discovering two prominent issues: the robotic arm’s movement path predictions are relatively accurate, but the objects’ positions and motions may still deviate from reality; models that perform well in the short term may degrade noticeably over a full operation sequence.

On October 16, the evaluation platform WorldOracle for action‑driven world models was officially opened to developers. The research team has already pre‑collected real operation processes, recording visuals and actions. Developers only need to use the initial frames and action sequences provided by the evaluation set to generate predicted videos and submit them to the platform. The platform then compares them with the corresponding real execution records for scoring, allowing one to determine whether the model accurately predicts the consequences of actions, and to pinpoint shortcomings and validate improvements.

• Paper: WorldOracle: Benchmarking Action-Conditioned World Models against Real-World Executions

• Paper link: https://arxiv.org/abs/2610.18826

Project homepage:https://worldoracle-benchmark.github.io/

1.First batch of scorecards: seven world models

Who can match the real machine?

01 The robotic arm moves correctly, but the object may not keep up

The process of the robotic arm reaching for the fruit and closing its gripper may resemble the real scene, but whether the fruit is lifted and where it moves afterward can be predicted incorrectly by the model. To differentiate these two types of performance, the research team separately compares the alignment of the robotic arm’s and the object’s positions and motions in the predicted videos with the actual recordings, and provides scores.

Spatial alignment score between the robotic arm and the manipulated object.

The results show that the robotic arm scores of all five evaluated models are higher than the object scores, meaning that even though the robotic arm in the scene follows the given motion fairly well, the object’s position and movement may still deviate from reality.IRASim’s robotic arm score is 78.8, while the object manipulation score is only 29.1; Cosmos 3 scores 89.4 and 45.4 on the two metrics. The other models also exhibit varying degrees of discrepancy.

02 Extending prediction horizon significantly degrades accuracy

Another challenge comes from time. When a robot completes a sequence of actions, it often requires multiple grasps, placements, and adjustments. The state of the object left by the previous interaction becomes the starting point for the next operation. Whether the model can continuously stay close to reality needs to be evaluated over the entire process.

WorldOracle evaluates both the first 30 seconds of the trajectory and the full long sequence; how does the model perform?

Comprehensive single-view score of the same model under two evaluation durations

The results show that, on the same set of robot teleoperation data, the single-view overall scores of all five models decreased. DreamDojo’s score dropped from 50.56 to 27.08, and Cosmos 3 fell from 53.99 to 26.60, both reductions close to half. GE‑Sim 2.0’s score in this test fell from 73.86 to 63.33, achieving the highest scores among the five models for both short and full-length sequences, and its decline on the long sequence was relatively small.

The relative ranking among models also changed: IRASim’s score in the first 30 seconds is lower than Ctrl-World, DreamDojo, and Cosmos 3, but its score for the full sequence is higher than those three models.

2. How does WorldOracle generate questions and score them?

01 Here, accurate failure prediction can also score high

WorldOracle’s evaluation starts from pre-collected real operation logs: provide the model with initial frames and action sequences, let the model generate predicted videos, and then compare them with the actual execution of the same set of actions.

The model needs to predict, including catching empty, dropping, and sprinkling in real operations.Even if the operation fails in reality, accurately predicting this failure is still a correct prediction.

Failure cases thus become an important class of test samples: they can verify whether the model follows the given action and its consequences, or whether it rewrites the process into a common successful version.

The evaluation covers the ZhiYuan G2 and YAM robot bodies, as well as first‑person human hand operation scenarios. Four task types—fruit picking, hanging a towel, pouring water, and opening/closing a drawer—involve interactions with rigid bodies, flexible objects, fluids, and articulated objects, respectively. The specific task coverage for each body follows the evaluation set.

In this context, the human‑hand scenario targets world models that can receive hand‑motion inputs, testing whether the model can predict subsequent object changes and interaction outcomes based on recorded hand‑pose sequences. Models equipped with the corresponding first‑person world modeling capability undergo zero‑shot evaluation in this scenario, without using any additional human‑hand adaptation data provided by the platform.

Real execution screenshots of the four types of tasks

The data includes two sources: one collected by humans completing tasks, including teleoperated robot trajectories and human hand operation trajectories; the other consists of robot policy rollout trajectories, generated by the policy controlling the real robot.

When the robot operates autonomously, pauses, jitter, and motion corrections that differ from manual operation may occur. Including this type of data helps verify whether the world model can still predict accurately when faced with actions from different sources.

The platform sets three tracks:

• WorldOracle-Core: Evaluates the predictive performance for the first 30 seconds of manually collected trajectories.

• WorldOracle-Policy: Evaluates the model’s predictive performance for the robot’s policy executing the entire process autonomously.

• WorldOracle-Long: Evaluates the sustained predictive performance in the full long sequences collected manually.

</ul




In the current embodied world model evaluation benchmarks, WorldOracle is the only benchmark that simultaneously covers action sequence inputs, real execution data, failure cases, long‑term interactions, multiple ontologies, multiple viewpoints, and autonomously executed policy trajectories.




02 Decompose prediction errors: consider both outcomes and processes




The robotic arm may be in a similar position, yet the fruit might not be grasped; the final states can look alike while the intermediate motion paths differ. A single score struggles to capture these nuances.




WorldOracle checks forecasts with four complementary metrics:




• Physical State Correctness: Verify that the object’s position, grasp state, liquid transfer, etc., at the end of each interaction match the real-world outcome.

• Interaction Space Alignment: Check whether the positions and movement paths of objects, robotic arms, or human hands are consistent with the real process.

• Fidelity of Key Interaction Areas: Focus on areas such as objects, grippers, and human hands, combining pixel, perceptual, and semantic similarity to reduce large‑area background interference in scoring.

• Multi‑view Action Consistency: For models that support multi‑view output, check whether the robot motions in each synchronized camera view align with the actual execution.









The first three metrics are equally weighted to form a single‑view composite score; after adding the fourth metric in multi‑view evaluation, they are equally weighted again. The sub‑scores are retained, allowing developers to distinguish issues in state prediction, motion trajectory, and local details.




To validate the reasonableness of the metrics, the study invited 119 participants to compare 128 pairs of predicted videos, finding that the four metrics’ judgments of prediction quality agreed with human choices at a rate of 88.1%–93.2%.




Of course, this evaluation also has clear boundaries: it measures the degree of match between a prediction and a single real execution record. For fluids, flexible objects, and random contacts, the same action may produce multiple reasonable outcomes, and a single reference cannot fully describe these possibilities.




The training support varies across different ontologies: G2 uses a specially collected training set, YAM uses public data, and the manual scenario employs zero-shot evaluation. Therefore, the results from the three scenarios should not be directly used to assess ontology difficulty or cross-ontology generalization capability.




3. Test Center Open Today




Open submissions on the platform to provide concrete feedback for model iteration




WorldOracle supports participation of open-weight and closed-source models. After developers complete registration and verification, they can select a track based on the ontology or operational scenario supported by the model, generate and upload prediction videos according to the submission guidelines, and view overall scores, component results, and comparison outcomes.




For model iteration, this feedback can be applied to specific issues: Does a single improvement make object motion more accurate? Does it reduce false judgments of failed operations? After extending the prediction horizon, does the improvement still hold?




Is the future generated by world models reliable? It must be tested against real execution. WorldOracle turns this testing into unified tasks, data, and scoring methods, providing developers with a basis to compare models, identify errors, and validate improvements.




• Project homepage:https://worldoracle-benchmark.github.io/

• Submission guide:https://worldoracle-benchmark.github.io/submission/







© THE END 

For reprints, please contact this public account for permission

Submit articles or request coverage: liyazhou@jiqizhixin.com