Do robots really need a “world model”?

Do robots really need a “world model”?

From September 20 to 21, the PAIR Conference 2026 with the theme “Frontiers of Next-Generation Physical Intelligence” was held in Shanghai. The conference was organized by the Shanghai Institute of Physical Intelligence and Robotics, and invited numerous experts and scholars from universities, research institutions, and industry to discuss topics such as next-generation intelligent computing systems, next-generation intelligent carriers, next-generation scientific discovery, and next-generation computing paradigms.

At the “Next‑Generation Intelligent Carrier” special forum, the reports and discussions centered on how robots perceive the environment, perform tasks, and continuously learn. The final round‑table, titled “From ‘Understanding’ to ‘Doing’: Embodied Intelligence Breakthroughs in the Physical AI Era,” tackled a concrete question: Do robots necessarily need a world model?

The round‑table discussion was chaired by Associate Professor Li Shuguang from the Department of Mechanical Engineering at Tsinghua University, with three researchers joining the dialogue, namely Associate Professor Sui Yanan from the School of Aerospace Engineering at Tsinghua University, Assistant Professor Ban Yutong from the Pujiang International College of Shanghai Jiao Tong University, and Assistant Professor Du Tao from the Institute of Interdisciplinary Information at Tsinghua University.

Photo | (Left to right) Li Shuguang, Sui Yanan, Ban Yutong and Du Tao having a round‑table discussion (Source: On‑site photo)

When a robot picks up a cup, it must locate it and control its movements to avoid knocking over nearby objects. One approach in world model research is to have the robot predict the consequences of its actions in advance before deciding how to act. This could reduce trial-and-error in real environments, but it also requires additional data and computation.

When Li Shuguang asked whether world models are a “necessary component,” the three guests gave answers each with a different emphasis.Their disagreements stem from the research subjects they each face, as well as from different understandings of “world models.”

The following content is compiled from the round‑table discussion.

Sui Yanan: Understanding the world may not require complex external models

Sui Yanan first places the problem on the robot itself. It must have some kind of “prior,” that is, pre‑existing knowledge about itself and the environment before acting.But how specific this knowledge should be and in what form it should be retained to help the robot complete tasks has no unified answer yet. His inclination is: “Maybe we don’t need a complex external model.”

Take picking up a cup as an example: the robot needs to know where the manipulator can reach, where the cup is located, and how to grasp it so it doesn’t slip. Whether every object in the room has been finely reconstructed in appearance may not be valuable for this particular operation. The amount of detail a model should retain must be determined by the task.

This judgment is related to his research subjects. Su Yanan’s team studies how human nerves, muscles, and skeletons work together to control movement, and also how such bodies interact with robots, wearable devices, and the external environment. For them, the “world” cannot be without humans, nor can a single human-shaped form replace the human motor system.

There is still a lot to fill in between static structure and dynamic motion. Anatomy can describe the positions of muscles and bones, but when a person raises an arm, takes a step, or interacts with a device, an embodied human model must also explain how the body moves and how forces are applied. To make assistive devices coordinate with human actions, these dynamic relationships directly affect control performance and deserve further refinement.

He explained that muscle and bone models are now relatively clear, and the team is gradually incorporating the nervous system. Peripheral motor and sensory nerves can already be described with increasing detail; when it comes to the central nervous system, the difficulty remains significant. Consequently, human motion models need to be advanced layer by layer.

The interface where the body contacts the environment is also something he focuses on. A purely rigid skeletal structure can hardly cover all situations in real interaction; skin and soft tissue also need to be incorporated into the model. A system that combines rigidity and flexibility can move agilely and better cooperate with the physical environment.

When discussing ways to represent world models, Sui Yanan distinguishes two approaches: one abstracts the world, preserving important states and relationships; the other attempts a more concrete simulation of humans, machines, and the environment. In recent years, advances in graphics, computer vision, and related fields have made the latter approach more feasible. However, whether the added detail actually helps robots accomplish tasks still requires case‑by‑case evaluation.

When the discussion extends to self‑evolving systems, he also reminds that “evolution” is a neutral term. The host asks whether more useful hands could evolve in the future; he thinks it’s possible, but the environment itself is also changing. A structure that works well under current conditions may not be as effective under different conditions. Body structure, control methods, and the environment need to be considered together, and “evolution” will not progress toward a single fixed form.

Ban Yutong: World models are valuable, but not necessarily optimal

Ban Yutong’s answer to “whether it is necessary” is more straightforward: “Not necessarily.” In his view, the definition must be clarified before the discussion. The world models referred to by different researchers may point to different things, and these works should not be judged as the same technology.

He is more focused on predicting the subsequent outcome based on the current action: if the robot pushes forward a bit, how will the object move; if a different grasp pose is used, what will happen next. The prediction can be expressed as video, 3D scenes, or other forms; the key is to link “what action to take” with “what consequence will follow.”

Researchers hope the model will learn physical laws during this process. But Ban Yutong points out that making the predicted frames look like real images, or making the 3D results close to observations, does not guarantee that the model truly understands motion and friction. With more data, performance may improve; whether there are better ways to learn these physical relationships remains an open question.

This retention is related to his manipulation research: whether it is a gripper or a dexterous hand, grasping and moving objects always involves contact, and contact entails forces and friction. An object may appear unchanged, but if the friction conditions on its surface change, the original action may no longer be effective. Robots need to cope with these differences that can alter the outcome of manipulation.

In terms of concrete algorithms, Ban Yutong’s team focuses on the design of action models, including how to reason faster and how to handle longer tasks. One piece of work requires the robot to press a button three times in a row, and the model must keep track of how many times it has already pressed. The team added a mechanism for the action model to process historical information, enabling it to better accomplish such tasks. What needs improvement here is the memory of task progress, not just the prediction of the next frame.

Surgical robot research has introduced another set of requirements. The team is trying to enable a dual‑arm robot to perform tasks such as needle handling and suturing, some of which stem from surgeons’ skill training. In these scenarios, safety must become an explicit step. Ban Yutong emphasizes that the system must set explicit safety constraints for critical tissues and blood vessels, rather than merely expecting the robot to avoid risks on its own while completing the task.

Prediction can provide a basis for action, while safety constraints dictate which actions should not be executed. Even if an action helps approach the goal, it must be restricted once it crosses a safety boundary. This task requirement makes “what the model can predict” and “what the overall system permits the robot to do” separate issues that need to be addressed individually.

Regarding how robots can improve themselves in the future, Ban Yutong proposes a concept: As the underlying operation models become more reliable, a more complete task execution framework can be built on top of them, with other models supervising the execution results to help robots correct errors. He also noted that there is not much work of this kind currently, an important reason being that the capabilities of the underlying models are still insufficient, and they have not yet provided sufficiently usable tools.

From these studies and proposals, improving a robot’s operational capability requires simultaneously addressing concrete issues such as memory, safety, and feedback. A world model might provide the system with better predictive foundations, but how much benefit it actually brings depends on how these predictions are incorporated into real-world operation.

Du Tao: The hardest part of simulation is the “last mile”

Du Tao’s view is different; he does not directly judge whether a world model is necessary, but chooses to start from the simulation he is familiar with: Building an environment in a computer and making that environment sufficient to guide a real robot are two tasks of different difficulty.

A simulator built according to physical laws has explicit equations and parameters, allowing one to trace how motion results are computed. Du Tao noted that the graphics community has also discussed whether such simulations can be understood as a type of world model. If a world model is seen as a description of the environment and its changes, then traditional physical simulation already fulfills part of that role.

His team does a lot of research on fluid and soft-body simulation. These simulations involve complex computational systems, requiring both accurate calculation of motion and sufficiently fast computation speed. The team studies how to perform parallel computing on graphics processors, how to design more efficient solvers, and also experiments with incorporating deep learning into the solving process. Whether a simulation is usable inherently requires repeated trade‑offs between accuracy and computational cost.

What truly consumes effort is the gap between simulation and the real world. Du Tao’s experience is that producing a roughly accurate simulation at relatively low cost is not difficult, but bridging the “last mile” can cost far more than expected. For example, formulas can describe how friction affects motion, but the friction parameters of the material at hand and the tiny errors of the device still need to be measured and calibrated. This work involves algorithms as well as extensive engineering and real‑hardware testing.

Through “differentiable simulation,” computers can also estimate how tiny changes in parameters affect outcomes, thereby optimizing actions or designs. Du Tao admits that whether these changes can be computed and whether they actually bring benefits to specific tasks are two separate issues. Writing such a framework technically is different from implementing it well in engineering. Whether the method is worth adopting still depends on whether it can improve the results of concrete tasks.

In his view, for a system to keep improving, the focus should be on building a self‑operating workflow. For the team’s research on non‑traditional configuration robots, the first hurdle may lie in hardware—for example, whether the designed structure can be assembled and tested reliably. At the same time, sufficient data are needed later to feed test results into the next round of improvements.

Real interaction data involving soft bodies and fluids are not as abundant as text and images, so researchers often have to collect and clean them themselves. If these steps must be continuously performed by humans, merely updating the model repeatedly cannot make the entire research workflow run autonomously. Under such conditions, computation and simulation remain essential support, and even if simulations are imperfect, they must still be used.

However, Du Tao does not treat this choice as the final answer.He believes that simulations built solely on known physical laws are insufficient to handle the complex, hard-to-describe factors in the real world.He hopes a model can fill this gap; it might be a world model or come from another approach. Whether the new model can truly bridge these gaps still needs to be tested together with real systems.

Sui Yanan’s reservation about model complexity, and Ban Yutong’s question about whether a world model is optimal, both leave room for choices in how robots understand the environment: a robot needs a basic understanding of the world, which does not necessarily mean building a complex predictive model first. Du Tao’s reminder about the “last mile” of simulation adds a practical constraint to this choice. How detailed a model needs to be depends not only on whether it helps the robot accomplish its tasks, but also on how much measurement, computation, and engineering effort is required to narrow the gap between it and the real world.

Returning to the original question, if the goal is merely to perform specific actions, a robot may not need an independent world model. The potential of a world model lies in its ability to let acquired experience continue to be useful in new tasks and new environments, reducing costly real‑world trial and error. Whether it can become a key component of a general‑purpose robot ultimately depends on whether these predictions allow the robot to rely less on human intervention while accomplishing tasks more reliably.

Operations/Typesetting: He Chenlong

Note: Cover/featured image generated with AI assistance