Chatting with a Robotics‑LLM researcher: Will LLMs “devour” robot models?
Chatting with a Robotic×LLM researcher: Will LLMs “eat up” robot models?
Summary
Astra, Opus and other LLMs are impacting robotics far beyond many researchers’ expectations: LLMs are now starting to serve as the robot’s “small brain.” In the past, a common
1.Astra and other LLMs have performed in robot control far beyond many robotic researchers’ expectations: although LLMs have already helped robots understand tasks and devise plans, the most noteworthy change this time is that LLMs are beginning to generate robot actions directly. In other words,the capability of LLMs is expanding from the “brain” (System 2) that understands tasks and creates plans to the “cerebellum” (System 1) responsible for action generation.
2.Taking Astra as an example, although it is not a model specifically trained for robot manipulation, it can generate control commands for robot position, pose, or joint‑level based on visual input. It can also flexibly generate dual‑hand operation trajectories according to the current scene, rather than merely reproducing a fixed trajectory.
3.In the RoboDojo simulation benchmark, Astra achieved an average score of 28.97 (out of 100) and an average success rate of 22.48% when the inference intensity was set to medium and a simple zero‑shot configuration was used, surpassing most of the VLA and WAM robot foundation models participating at the time.
In this evaluation, the score reflects how much of the task was completed, while the success rate requires the task to be completed fully.
4.However, a score of 28.97 is not necessarily the upper limit of Astra’s current capabilities. To compare different models fairly, RoboDojo’s initial tests used a relatively simple harness that basically let the model output actions directly. If the model could check results after actions, analyze errors, and then adjust the next action, the task performance could improve further.
5.Some view that, by the end of 2026, a model will achieve near‑perfect scores in the RoboDojo evaluation.
6.Astra’s another capability that amazes robotic researchers is cross‑ontology generalization:that is, the same model can gradually adapt to robots with different structures based on the robot’s own body information and execution feedback.
•After a tester provides Astra with information about a custom robot’s appearance, degrees of freedom, and the type of gripper or hand, Astra operates the robot. Astra initially makes errors, but after a few retries and reflections, the control gradually becomes more accurate;
•Another tester lets Astra directly control the joints of an unfamiliar robot. The first few frames are chaotic, but as attempts continue, the control becomes increasingly precise and the motions smoother.
7.But some researchers argue that Astra’s performance does not signify a qualitative change in robot capabilities, because the model’s behavior on robots is more like a natural extension of its existing coding and gaming abilities. Considering that LLMs have long been able to correctly understand task intent, this round of improvement may mainly come from more accurate 3D position estimation.
•For example, during testing, the model spends a lot of time inferring pixel coordinates, estimating camera parameters, and then calculating the 3D position of objects, but the resulting actions mainly involve changing spatial positions and rarely involve pose adjustments.
8.However, the inference speed being too slow is one of the most immediate issues with current LLM solutions of this type. Moreover, the actions that Astra can handle now are also very simple, and it is currently difficult to directly extend to dexterous hand or full-body robot control tasks that require coordinating dozens or even hundreds of degrees of freedom.
9.In addition, the impact of LLMs on robotics has expanded from direct motion generation to the model training paradigm.For example, many researchers are now exploring how to borrow the development trajectory of LLMs so that robot models can also possess capabilities such as in‑context learning; LLMs have also dramatically shortened the time required to build simulation environments, which has led some researchers to regain confidence in the simulation approach.
Insight 02
“Big‑and‑Small Brain” model: integration or continued division of labor?
10.As LLMs can both perform task planning and directly generate actions,whether robots still need a separate “cerebellum” model is the main point of contention among robotics researchers today.
11.Proponents of brain‑size integration argue that task understanding and action correction both require world knowledge, and isolated small models may not have enough capacity to master these abilities. Since LLMs can already comprehend tasks and correct errors based on feedback, System 1 and System 2 could potentially be handled by the same model.
•Some researchers also anticipate that System 1 and System 2 will continue to coexist in the short term, may merge in the future, but there is currently no consensus on their long‑term division of labor.
12.Those who advocate keeping separate robot models argue that a robot model must not only generate correct actions but also respond to environmental changes promptly; if a general model’s inference speed cannot keep up, low‑latency action models remain valuable.
•Dynamic task scenarios especially require the model to provide the next action within a limited time.For example, when objects on a desk are stationary, the robot can pause its actions and wait for the model to finish inference; however, if the robot is performing tasks like pouring water, the water will keep flowing, and the robot must adjust its actions promptly.
13.In addition, fine contact operations require faster responses; this type of low‑level control is referred to as System 0. For example, the contact point between a robotic arm and an object is often visually occluded, but in practice a deviation of just a few millimeters can cause the object to slip, so the robotic arm must quickly adjust based on feedback from tactile and other sensors.
•Take peeling a cucumber as an example: a person can first decide to “remove the skin,” but once the peeler contacts the surface, the hand must quickly adjust force and position, moving along the cucumber’s surface. If every tiny contact requires the model to perform complex inference again, the robot’s movements will be too slow.
•Some researchers estimate that this kind of low‑level control needs to respond dozens of times per second, and cloud communication combined with model inference could introduce latency of one to two hundred milliseconds or even more. Therefore,even if System 1 and System 2 are integrated, System 0 may still need to run locally on the robot.
14.For example, York, co‑founder of Dyna, recently emphasized the requirements that real‑time control imposes on deployment methods.In his vision, System 2 can provide low‑frequency guidance through language, while System 1 and System 0 need to respond promptly. In addition to model inference speed, network fluctuations also affect execution, so he wants both of these layers to run locally on the robot, leaving only System 2 with cloud access.
15.Regarding how these models and control layers are divided, some researchers believe that System 0, 1, 2 have no fixed boundaries, and how the models share tasks depends on the action scope, response speed requirements, and the required model size. Tasks with relatively fixed input‑action mappings and needing fast response can be assigned to small models; tasks with a broader action range and more complex decisions require more knowledge and computation.
Insight 03
In-Context Learning: Which Methods from LLMs Can Be Applied to Robot Training?
16.In‑context learning is an important direction for robots to improve generalization: the model can use human demonstrations and its own execution feedback within the context to adjust actions, without needing to retrain the model for the current task.
17.However, some teams’ experiments have found that directly adding execution history and reflective capabilities to VLA does not automatically provide error‑correction or adjustment capabilities.
•A team tried to have VLA leverage historical information; as a result, although the error decreased during model training and validation, the model did not perform better upon deployment and sometimes even got worse. One possible reason for this phenomenon is that the way historical information was incorporated actually interfered with the capabilities the model had learned during pre‑training.
•Another team adapted VLA based on Qwen, enabling the robot to place five or six snacks and drinks into their respective zones in a specified order. During training, the team annotated the complete plan, whether each step was completed, and what to retry after a failure. In testing, the model could recite the plan and identify which step failed, but it still repeatedly performed incorrect actions. Compared with a standard VLA that does not use history, its success rate showed almost no improvement.
18.In fact, for reflective ability to improve robot performance, the model needs to translate language feedback into concrete action adjustments.A researcher tried to annotate language feedback and actions more finely. For example, when a drawer isn’t fully opened, instead of just writing “failed, try again,” they also describe “the drawer isn’t opened enough, pull it a bit further,” and pair it with the corresponding action. As a result, compared with only adding CoT annotations to the training data, adding CoT on top of fine-grained annotations yields a model with better performance.
19.But does this robot model’s in-context learning ability have to rely on specialized failure and recovery data for training,Researchers have differing opinions.
•In a demo of Generalist, the robot grabbed too high the first time and lowered its position the second time. Some argue that this suggests the training data may already include similar failure and recovery demonstrations, and the model learned this adjustment from them.
•Another viewpoint holds that the model’s error‑correction ability may also be transferred from other tasks.
1) Considering that the proportion of erroneous examples in the existing collected data is low, human mistakes may not cover the errors that actually occur during robot execution. Relying solely on this type of data makes it difficult to train a correction capability that can handle a variety of failure scenarios.
2) The robot model can retain the LLM’s input‑output paradigm, which also gives the robot the opportunity to leverage the adjustment capabilities already present in LLMs. For example, the model learns to correct errors based on execution results during multi‑turn coding training, and learns to adjust subsequent actions based on visual feedback in computer‑use tasks. Some researchers believe that we are already seeing partial transfer of such abilities.
20.Even if the model has in‑context learning capability, whether to incorporate action execution history and what information to include in the context during the model’s actual task execution ultimately depends on the specific task.
•Based on estimates, about 80% of robot tasks can decide the next step just by observing the current state, and handling these tasks already covers a large market.
•Long tasks where steps depend on each other, and tasks requiring high‑precision operations, are more likely to benefit from incorporating action execution history. For example, the model can combine previous actions and outcomes to infer how the robot moves and how the environment responds, then adjust the next action accordingly.
•Additionally, task instructions also serve as useful context. For example, when a robot encounters furniture it hasn’t seen during assembly training, it may need to read the manual first.
21.Furthermore, as the robot’s context continuously accumulates, issues of action stability and reasoning latency become evident in long contexts.
•A team set the model’s context length to about 8K tokens in an experiment. As the execution history accumulated, when the context reached 7K‑8K tokens, the robot’s actions began to become chaotic.
•A researcher attempts to use test-time training to compress context, control latency, and also let the robot correct errors based on feedback.(ICL retains new information in the context, while TTT updates model parameters through training)However, after testing Astra, he found that Astra’s in-context learning capability already achieves many of the error‑correction effects he originally hoped to obtain via TTT.
Insight 04
Astra makes simulation great again
22.The impact of LLMs on the robotics simulation field has also resonated with researchers:LLMs have dramatically shortened the time required to set up simulation environments, which has even led some researchers to reconsider and favor the technical approach of scaling model training through simulation.
•Since Astra appeared, the simulation team can now set up parts of the evaluation environment that used to take months in just a few hours—or even minutes.
•Once the simulation environment is built, robots can repeatedly try different actions within it and, based on feedback, improve their action policies through RL. Some researchers even hope to incorporate this trial‑and‑error learning during the model’s pre‑training phase.
23.However, speeding up environment setup and making simulation more accurate actually solve different problems: ray tracing, 3D reconstruction, and Gaussian representation make the scene “seen” by the camera more realistic, but physical simulation must be computed precisely—how an object moves, deforms, and contacts after a hand applies an action.Visually it looks more like reality, but that doesn’t mean the same action will produce the same result in simulation and in the real world.
24.Which tasks simulation can cover still depends on how accurately the underlying physics engine can reproduce the physical processes in the tasks.Existing physics engines have not made a qualitative breakthrough compared to five or six years ago; simulated results for contact forces, haptics, etc., can still differ from reality by an order of magnitude. Therefore, some view that it is better to first scale up model training on tasks where simulation is already relatively mature, such as grasping and transporting objects of fixed shapes.
•To recreate a real object in simulation, you need to know its movable structure and physical properties. A lab once purchased several thousand jointed items such as drawers, scanned each one, segmented the movable parts, and then established opening‑closing relationships. Even after completing this work, only partial attributes like mass were measured, and information such as opening resistance remains incomplete.
•Although game engines have accumulated a large amount of ready-made assets, they still struggle to directly provide the physical feedback needed for fine-grained manipulation. Because games need to run in real time, they often skip cumbersome processes such as dressing or eating, or present them with pre-made animations. These assets can help models learn basic spatial relationships like near‑big and far‑small, but they cannot directly supply the physical feedback required for precise operations.
25.For robotics teams, improving simulation fidelity often raises training costs. The more detailed the simulation of complex contacts, deformable objects, and so on, the slower the computation and the higher the cost, which means fewer action attempts can be made in simulation under the same budget.When the cost becomes too high, directly collecting real-world data may even be more economical.
26.To reduce this portion of cost, you can also use GPU acceleration, increase parallelism, or use AI to optimize low‑level code, but some researchers believe that the industry as a whole still lacks sufficient talent and funding for such low‑level optimizations.
27.Some teams also try using world models to predict the state changes caused by actions, replacing part of the simulation calculations to reduce training costs.
•For example, SuperMimic trains a world model with simulated data, allowing it to predict how a robot’s body state will change after executing a particular action, and then uses these predictions to train a full-body motion control policy for humanoid robots. However, researchers note that although SuperMimic involves complex motions such as flips and side rolls, the simulated subject is still a robot composed of rigid components and joints.
28.However, the capability of world models ultimately depends on the training data. If the data only come from existing simulators, the physical variations they can learn are also limited by the simulators’ capabilities. Moreover, even when real data are introduced to supplement contacts and deformations that simulators struggle to cover, that real data may still fail to capture complete information from the real world. For example, when photographing clothing with a camera, the surface condition of the garment can be recorded, but hidden folds underneath cannot be captured, making it difficult for the model to reconstruct the garment’s full state from those images alone.
29.Another way to reduce training costs isto teach the model to adapt to the differences between simulation and reality, thereby reducing reliance on high‑fidelity simulations.Some viewpoints suggest first letting the model learn to observe, act, and correct errors in coarse‑grained simulations, and then adapt to the real world with only a few attempts.
•But this idea has a premise: the effect of the same action in simulation and reality must not differ too much. If the simulation results consistently deviate significantly from reality, the model will find it difficult to adapt to the environment with only a few trial-and-errors.
•In fact, the required simulation accuracy depends on the specific task. For example, cleaning fixed‑shaped objects in a room can tolerate coarse simulation for some operations, but fine‑contact tasks such as pulling a zipper require higher accuracy.
30.Furthermore, researchers hope that the model can accumulate experience and improve its capabilities through continuous interaction with the environment. In the robotics field, the concept of Physical RSI is: first let the robot perform tasks in the real world, use the data obtained during execution to train the model in simulation, then deploy the updated model back to reality, iterating repeatedly.
Insight 05
What kind of data is useful for robot training?
31.In addition to model capability and training methods, the training data also has a huge impact on the accuracy of robot motions. How much improvement an hour of data can bring depends on which tasks and motions are included in the data, as well as how closely the demonstration data matches the robot’s actual operation.
32.Ego, UMI, and real‑robot data each have their pros and cons; overall, the closer the data is to the robot’s actual executed motions, the harder it is to collect it at low cost and large scale.
33.Some teams use ego data to let models learn how to plan tasks and understand spatial relationships, but to enable robots to master how much force to apply during manipulation and how to perceive contact, they still need to collect force and tactile data using devices such as sensor‑equipped gloves.
34.Those who are optimistic about Ego’s long‑term value believe that future robot hands and bodies will become more human‑like, and with better data cleaning, quality control, and task diversity, the model’s ability to learn actions from human videos may improve.
35.For real-device data, this type of data is closer to the robot’s actual actions, but it is more costly and harder to scale up collection.
•，real‑device collection should be prioritized for tasks that are difficult for simulation to model accurately, such as fluid manipulation, folding clothes, and similar scenarios.
•Real-device collection is also limited by scene coverage: in some non‑home scenarios, robots have a harder time entering, while UMI can collect more easily; this difference is relatively small in home scenarios.
UMI data demand is rapidly increasing
36.Recently, the demand for UMI data and related hardware has been increasing.
•Some teams, after training with a large amount of Ego data, saw improvements in model capability, but the motions were still not accurate enough, so they turned to using UMI.
•For gripper operations, the motions captured by UMI are closer to the robot’s actual execution. A researcher estimates that the training effect of 50 hours of Ego video may be equivalent to only 5 hours of UMI data.
•Whether a team needs to purchase UMI also depends on whether the team has its own robot: teams with a fixed platform usually collect UMI data around their own grippers or hands; teams without a specific platform rely more on external suppliers.
37.However, whether UMI can become a long‑term data roadmap remains disputed in the market.
•Those who are optimistic about UMI believe that handheld grippers are more convenient for data collection than real devices, can provide motion information missing from ordinary video, and can also add tactile and other sensors. Even if companies collecting UMI data later adjust their data pipeline, the existing UMI data will still be valuable.
Data Scaling vs Data Recipe
38.In addition to which tasks the data itself includes, the data’s selection, annotation, and training ratio also affect the model’s final performance.
•Some low-cost purchased Ego data may contain a large amount of repetitive wiping and cleaning operations. Without further filtering and detailed annotation, the total hours of data may appear long, but the new experiences they can provide to the model remain limited.
•Even when data from different sources, formats, and ontologies are unified, you still need to determinehow much of each type of data to use and at which stage to add it.
39.Should we first increase the data volume or first determine the effective ratio? Researchers remain divided.
•Those who support scaling the data first argue that the current dataset is still far from sufficient and that expanding collection and training scale should be prioritized. For example, Dyna used over 1 million hours of ego data to pre‑train Dyna‑2.
•Advocates who first define a data recipe argue that, since the data volume has already increased dramatically, we should first verify how Ego, UMI, and real-device data combine most effectively, and then scale collection according to the optimal ratios.Especially for AI labs—teams that already have models and large data repositories—understanding which capabilities the new data can augment and how the new data should be mixed with existing data for training is particularly crucial.The team’s collection hardware, training experience, and compute budget also determine how large a scale they can collect data at and how many training iterations they can run.
Insight 06
Opportunities for Nimble Hands: Tools in This Society Are Designed for Human Hands
40.When collecting data and validating model performance, the robot team can freely choose to use a gripper or a dexterous hand according to task requirements, but in actual deployment the robot must complete the entire workflow, and the advantage of the dexterous hand becomes evident.
•Take making a milk tea as an example: even if nine out of ten steps can be done with a gripper, as long as one step requires a dexterous hand, a robot equipped only with a gripper cannot independently complete a full cup of milk tea.
•Work in a chemistry lab often involves various tasks, such as retrieving reagents from narrow cabinets or handling thin wires and slender containers. Overall,the tools used in experiments are mostly designed for human hands; even if a gripper can perform some actions, it may not be able to safely grasp and use these tools. Redesigning tools for the gripper would also increase modification costs.
41.However, whether the operational advantages of dexterous hands will quickly generate large‑scale demand remains disputed among researchers; the key lies in whether industrial scenarios actually need dexterous hands.
•One view holds that standardized factory tasks can be handled with simple grippers; for dexterous hands to be deployed at scale, household robots must become widespread first, and it will still take a long time for robots to enter homes, so large‑scale demand for dexterous hands is unlikely in the short term.
•Another perspective holds that some industrial tasks now demand both high load capacity and dexterous manipulation. For example, when processing seafood, a robot must securely grasp heavy, slippery items while also flipping them and aligning them with tools—tasks that standard grippers struggle to perform reliably. Although grippers can be customized for fixed processes, if a robot constantly requires gripper modifications or can only execute fixed motions, customers may opt for dedicated machines instead.
42.Whether the dexterous hand can operate reliably over the long term is also an important factor affecting deployment requirements.Some users have experienced the dexterous hand failing after just two weeks of use, or even breaking the next day.
43.Beyond hardware,whether the model can coordinate multiple fingers also affects the performance of dexterous hands.Some researchers note that in demos where general-purpose models control dexterous hands, the fingers still behave like simple grippers, opening and closing, and there are cases of control failures or suboptimal performance. In other words, although the model has been trained on a large amount of human manipulation video, due to factors such as the high degrees of freedom in dexterous hand joints, robots still cannot make effective use of dexterous hands.
44.To balance dexterity, stability, and cost, the three‑finger hand is seen as a compromise solution. Proponents argue that it is more flexible than a two‑finger gripper, the fingers can be made thicker and more stable than those of a five‑finger hand, and the cost may also be lower. However, the three‑finger hand has the following issues:
•Data collection: Some worry that human five‑finger motions are hard to map directly to a three‑finger hand, but supporters argue that you can have people operate in a way that a three‑finger hand can perform during data collection, thus obtaining data that matches the hardware.
•R&D investment: One practitioner said that the team already has a two‑finger gripper, and new R&D resources are often prioritized for the five‑finger dexterous hand, making it difficult to form a separate team to develop a three‑finger hand. Even if a three‑finger hand demo is produced, further continuous optimization still requires substantial investment.
Insight 07
Under the impact of LLMs, embodied companies’ advantage lies in integrated data, hardware, and deployment
45.In the wake of LLMs, embodied companies need to reassess the focus of their R&D investment:Should they continue to push the intelligence ceiling of robot models, or first optimize the execution speed and cost of robots for specific tasks?In other words, should their System 1 model become an efficient controller leaning toward System 0, or a general decision‑making model leaning toward System 2.
46.Supporters of continued model intelligence improvement argue that the physical world presents a wide variety of tasks, and training solely on coding-like tasks may not solve all robot control problems. If sufficiently strong open‑source LLMs remain available, embodied companies can build on these models to conduct further pre‑training or post‑training specifically for robotic tasks. As funding, compute, and talent increase, embodied companies may also choose to train their own foundational models from scratch.
•Moreover, improving the robot model’s capabilities may also reduce the adaptation cost for new tasks. York said in the interview that from Dyna-1 to Dyna-2, as the pre‑trained model became stronger, the time required for task fine‑tuning shrank from months to days. At the same time, Dyna has refined its data collection, filtering, and deployment tools, enabling the operations team to handle more on‑site work and lessening reliance on developers.
47.The other side believes,In-house robot models may not necessarily surpass general models in capability; embodied companies should instead leverage the existing capabilities of general models to enable robots to accomplish specific tasks in real deployments faster and at lower cost.A researcher speculates: if robot data is incorporated into the training of a general model, even if the added data amounts to only 1% or even 0.1% of the original training data, the model could quickly learn robot operations by leveraging its existing knowledge and multimodal abilities.
48.One potential approach to using LLMs to improve robot task execution efficiency is for embodied companies to distill a small model specifically for robot control from Astra. Specifically, the team must first have a real robot or simulator execute actions generated by Astra, and then train the robot model using the execution logs. This takes time and requires investment in hardware and simulation facilities, and some argue that collecting this training data may not actually be cheaper than gathering data directly.
49.Even though embodied companies’ small robot models infer faster and have lower operating costs,this speed and cost advantage may not be sustainable in the long term.Because computer use and voice interaction also require rapid responses, LLM vendors are motivated to improve inference speed.
50.Therefore, some researchers state that,embodied companies should, during this window before the acceleration of general model development, quickly apply efficient robot solutions to real-world operations.“It can be done technically” and “who can do it faster” are different questions.
51.When delivering to customers, embodied companies need to coordinate models and hardware, providing a complete robot solution that can perform real tasks.
•In scenarios such as folding clothes or towels, customers are more likely to pay for a complete solution rather than just calling a model API.
•In retail and industrial scenarios, some teams previously used traditional algorithms to plan robot actions. They have now switched to fine‑tuned π0.5 or proprietary models to generate motions, but after adopting the models, teams still need to perform on‑site tuning for each customer’s specific tasks before the robot can carry out real work.
•Overall, the amount of data to be collected and the amount of debugging required when entering a new scenario will affect the delivery cost for embodied companies; runtime stability and execution speed will affect how much work a robot can complete each day.
52.In addition, actual deployment is also a way for embodied companies to obtain model training data: just as LLM vendors can train models by feeding back user data from their own products, embodied companies can improve models and products by continuously recording the processes and outcomes of robots performing tasks. Moreoverdeployment can generate a scarce type of data: how robots fail and how they recover.
53.Therefore, for embodied companies, to obtain and leverage this feedback, end‑device hardware design, data collection, and customer deployment must be considered together: the end determines which motion data can be captured and executed, deployment verifies whether these motions can accomplish real tasks, and feeds back real data.
Further Reading
›
›
›
›
›
Get MirrorSpeed
Get MirrorSpeed