COCOMatrix Releases the CO01 Model: Teaching Robots to Learn While They Work

COCOMatrix releases the CO01 model: Learning by Doing, Teaching Robots Through Example

People learn by observing, trying, and receiving guidance; robots can do the same.COCO Matrix has released the robot model CO01 (pronounced /kuːl/), enabling robots to learn from these experiences as well.

A single demonstration, a failed attempt, a reminder, or even a target image can help CO01 decide its next step. The team calls this approach “experience prompting”: experiences can be inserted directly into the model’s context without retraining for the current task.

This reduces reliance on dedicated demonstration collection and fine‑tuning for new tasks, lowering adaptation and deployment costs. Users can also teach the robot through demonstrations and guidance. Failures, retries, and human corrections can serve as immediate references for the robot and, after organization, be used for subsequent training, allowing accumulated experience to continuously improve the model.

Hope CO01 becomes a very COOL robot model! Truly realizing the vision of the Coco Matrix: Robots for all, shaped by all!

Key points of this article:

• Faster onboarding for new tasks: Under the same pretrained model, a single demonstration of about 1 minute yields an average success rate of 52% on unseen tasks; fine‑tuning to a comparable level requires roughly 180 demonstrations, about 9 hours.

• Failure experience leads to quantifiable improvement: With the same five attempts, the real robot that retains its own history achieves a cumulative success rate of 84%, versus 56% without history.

• Robust Technical Support: Conversational training preserves the link between demonstration, failure, and correction, allowing the motion modelto adjust behavior using experience without updating weights. Interaction logs accumulated during deployment can be organized for subsequent training, enabling real‑world usage to continuously drive the data flywheel.

• Unified Perception Capability: A single set of weights supports depth, segmentation, optical flow, human pose, and other perception tasks, achieving better performance than specialized models on 5 out of 8 benchmarks, with depth estimation essentially on par, providing the perceptual foundation for robots to understand spatial relationships and motion changes in demonstrations.

• Technical Blog: Experience is the Prompt: Introducing CO01

• Blog URL: https://www.ccmtrx.com/research/co01

Table of Contents

1. From In-Context Learning to Experiential Prompting

2. Learning from Different Experiences

2.1 Demonstration ・ 2.2 Self-Practice ・ 2.3 Real-Time Guidance ・ 2.4 Cross-Body Skill Transfer

3. How Experience Becomes the Basis for Action

3.1 System 1 and System 2 ・ 3.2 Conversational Training ・ 3.3 Unified Visual Backbone

4. From Immediate Adaptation to Continuous Learning

1. From In-Context Learning to Experience Prompts

In the past, teaching a robot a new task often required human teleoperation to collect a large number of demonstrations before fine‑tuning the model. Switching to a different job could mean repeating the entire process.

In-Context Learning (ICL, contextual learning) enables the model to adapt to tasks using context without updating its weights. GPT‑3 demonstrated this capability in language tasks, and the robotics field has also begun exploring learning from demonstrations. CO01 further incorporates multiple experiences such as demonstrations, attempts, and real‑time guidance into action decision‑making.

Real‑world tasks are impossible to enumerate exhaustively. CO01 explores enabling new tasks to be taught on‑site, execution deviations to be corrected promptly, and past experience to be reused, providing technical support for addressing generalization challenges, reducing deployment costs, and promoting cross‑scenario replication.

Figure 1: From task instructions and single demonstrations to multi-source demonstrations, prior attempts, real‑time guidance, and target images, CO01 expands the contextual references available to the policy model.

 2. Learning from Different Experiences

2.1 Learning from Demonstrations: Adapting to New Tasks and New Scenes

CO01 can understand the goal and steps from a first‑person human view, side‑view video, or demonstrations by other robots, and find an approach that suits its own body. When wiring, it reused the hand‑over steps from the demonstration, while adapting the grasp method and motion path to its gripper.

Cable connection demo: the lower left shows a side view of a person, the main screen shows the robot operating autonomously, both at 2× speed.

Play the video of a person building blocks in reverse, and you get a demonstration of disassembling blocks. After CO01 watches it, it completes the disassembly following the reverse order.

Play the construction process in the original order at the lower left; the robot receives the reversed version and autonomously performs disassembly. All at 3× speed.

The demonstration can also combine existing actions. The model was trained only on single folds, yet after watching a person fold a towel twice, it sequentially completed two folds; when a person points to buttons one by one, it presses them in order.

Continuous folding: the lower left shows the human demonstration, the main screen shows the robot executing autonomously, both at 2× speed.

Follow the direction button, the robot executes autonomously at 4× speed.

To verify whether the robot used the demonstration, the team kept the scene and initial state unchanged, only altering the towel folding method or button order.

Left shows diagonal folding, right shows side folding; the lower‑left corner corresponds to the demonstration, both at 2× speed.

The demonstration order is shown in the upper right corner, and the robots on both sides execute according to their respective demonstrations, both at 1.5× speed.

Results show that 96% of button operations followed the demonstration order, and 92% of towel operations used the demonstrated folding method.

The team further altered the scene layout, objects, tools, and tasks to compare the effects of “providing a demonstration” versus “only giving verbal instructions,” testing each condition 100 times.

When switching to functionally similar alternative tools, a demonstration raises the success rate from 14% to 52%; when facing skills unseen during training, it increases from 6% to 43%.

Figure 2: The left panel starts from the original scene and sequentially tests a new layout, new object instances, and functional substitutes (S0–S3); the right panel sequentially tests known tasks, new skill combinations, and unseen skills (N0–N2). Each point represents 100 trials.

On another set of unseen tasks, the team compared ICL and fine‑tuning using the same pretrained model, with preparation time including demonstration collection and training.

ICL uses a single demonstration, about 1 minute, to achieve a 52% average success rate; fine‑tuning to reach a comparable level requires about 180 demonstrations and roughly 9 hours, a preparation‑time difference of about 460×. This means new tasks can first be quickly validated on‑site with a demonstration, and additional training can be added as needed.

Figure 3: Preparation time and success rate on unseen tasks. Time includes demonstration collection and training; the shading represents the 95% confidence interval; the ~460× is calculated using unrounded times. The fine‑tuning peak is 800 demonstrations, 26 hours, with a success rate of 66%.

2.2 Learning from One’s Own Attempts: Using Failure to Improve the Next Action

You only know whether a test tube fits into a hole after you try it. The robot that retained previous attempts tried the left and right sides, and on the third attempt placed the test tube into the blue rack; on the side without history, it tried twice on the same rack and the test tube fell.

Left shows retaining previous attempts, right shows not retaining history. Both sides operate autonomously at 2× speed.

The demo of the phone box also showed the same difference: the robot that retains history closed the box lid correctly on the fourth attempt; the side without history kept misaligning the lid each time.

Left shows retaining previous attempts, right shows not retaining history. Both sides operate autonomously at 4× speed.

The real-device evaluation selects tasks that frequently fail on the first attempt, with up to five attempts per task, keeping the task, scenario, and instructions unchanged. When retaining its own history, the cumulative success rate rises from  33% on the first try to 84% within five attempts; with the same five attempts but without retaining history, it is  56%, and using failure records from other tasks yields 58%. The related failure experience makes retries more effective.

Figure 4: (a) The same model with and without historical attempts during training, comparing action loss on multiple attempt records that were not part of training; with history, the loss is about a quarter lower and reaches the baseline optimum earlier. (b) Real-device cumulative success rate, each point based on 100 trials, with the shaded area representing the 95% confidence interval.

2.3 Learning from Real‑Time Guidance: Adjusting Current Actions to Meet New Requirements

The robot can still receive human guidance after it has started moving. The robot is delivering a cup to the cup rack; when a person places a coaster in the middle, it instantly changes the cup‑placement location.

The robot operates autonomously, receiving human guidance mid‑task. Actual speed.

A single instruction can become the basis for subsequent actions. The person first uses gestures to specify the pairing of cups and coasters, then scrambles the coaster positions; without new prompts, the robot still places them according to the previously taught relationship.

Left: gesture-specified pairing; Right: autonomous execution after scrambling the coasters, both at 4× speed.

Guidance provided during execution is retained in System 1’s context for reference in later actions; cross‑task preferences are stored by System 2 and re‑offered in subsequent tasks.

2.4 Learning from Other Robots: Cross‑Body Skill Transfer in Simulation

In simulation, a dual‑arm robot first sweeps debris into a dustpan, a wheeled robot uses that video as a demonstration to complete the task, and a humanoid robot then learns from the wheeled robot. The goal and steps remain the same, while each of the three robots grasps the tool in a different way.

Skill relay in simulation: the last two robots sequentially use the previous robot’s video as a demonstration, then execute autonomously at real speed.

3. How Experience Becomes the Basis for Action

3.1 System 1 and System 2: From Experience to Action

System 2 organizes experience. It is responsible for planning, memory, invoking tools, and monitoring progress, selecting demonstrations, prior attempts, or target images for the current subtask, and passing failure records to System 1 for reference in the next round.

System 1 generates actions. It combines reference experience, real‑time visuals, and robot state to make decisions, and continuously receives guidance during execution. During training, it predicts both future frames and actions; during deployment it outputs only actions.

Figure 5: System 2 organizes the experience needed for sub‑tasks, while System 1 generates actions based on real‑time observation. Both convey task instructions as well as demonstrations and interaction records.

The technical focus of CO01 is to enable the motion model to directly use demonstrations and interaction records, preserving operational details that are hard to describe in words.

3.2 Conversational Training: Preserving the Causes and Consequences of Success and Failure

CO01 stores demonstrations, failures, retries, and guidance within the same task as “sessions” in chronological order, training the model to use past experience to predict the next action. Single execution records also participate in training to retain existing skills.

Sessions come from dedicated data collection and existing video archives. Records from different sources are aligned by task and step, annotated with the help of a vision‑language model and manually reviewed; simulation and video generation further expand physical conditions, scenes, and viewpoints.

During training, relevant demonstrations and prior attempts are placed together, sometimes mixed with unrelated experience, so the model learns to discern what is worth referencing.

Figure 6: Targeting the third microwave reheating of noodles, with earlier attempts placed in front, as well as similar demonstrations provided by humans, other robots, and video models; occasionally unrelated tasks are added. During training, some metadata tags are randomly dropped, and the context length and experience combinations are varied.

3.3 Unified Visual Backbone: Understanding Space and Motion

The unified visual backbone is pretrained on first‑person human videos, recordings from various robots and handheld grippers, simulations, and filtered online videos; the team has collected and processed roughly 50,000 hours of data.

In addition to predicting future frames, the same backbone also learns depth, surface normals, segmentation, optical flow, as well as perception tasks such as human, hand, and camera motion, making manipulation‑relevant information easier for the model to exploit.

Perceptual capabilities of the visual backbone: depth, surface normals, optical flow, and segmentation of real-device footage, as well as hand skeletons, human skeletons, speech localization, and spatial localization. The first five frames come from the same real-device video, while the remaining frames are independent cases; hand and localization use the P10000 EMA selected results, presented separately from the benchmark evaluation below. Dynamic clips play at original speed, and after shorter clips finish they pause until the end of the current round.

Figure 7: Comparison of the same set of weights with a specialized model—five out of eight perception benchmarks lead, depth is roughly on par, and two lag behind. The raw errors for camera pose and optical flow are about 19% and 18% higher, respectively; the figure shows normalized scores, which do not correspond to the raw error change ratios.

The team also evaluated the impact of data scale and task diversity: As pre‑training data increases, the performance of ICL and fine‑tuning continuously improves; when ICL training covers 3,543 tasks, a single demonstration achieves a success rate of 53% on unseen tasks, compared to only 18% with just a language instruction. Expanding the scale and coverage of training experience helps boost the model’s ability to adapt to new tasks.

Figure 8: (a) Increasing pre‑training data, comparing 800 demonstration fine‑tuning with single‑demo contextual learning on a new task set; success rates at 50k hours are 87% and 53%, respectively. (b) Adding more types of contextual training tasks, comparing single‑demo with language‑only instructions. The horizontal axis is logarithmic, 0 baseline single column; fine‑tuning methods and evaluation rules remain consistent across scales.

4. From Instant Adaptation to Continuous Learning

In the next phase, the team will focus on reducing error accumulation in long tasks and improving the accuracy of CO01’s selection experience. Experience accumulated in one scenario can be applied to similar scenarios, further lowering deployment costs.

The team also plans to enable CO01 to proactively seek guidance: request demonstrations when uncertain, schedule practice after repeated failures, and draw on experiences from other robots’ logs.

Currently, new skills mainly adapt through in‑context learning, and cross‑task preferences are stored by System 2. COCO’s long‑term goal is to drive continuous model evolution through execution experience, enabling robots to share skills, accelerate learning and adaptation, ultimately serving thousands of industries and entering thousands of homes.

The COCO matrix is continuously seeking to further optimize CO01 partners, and welcomes researchers and engineers interested in In‑Context Learning, continual learning, multimodal representation learning, and real‑world experiments to join.

Reference Links

1. Brown, T. et al. Language Models are Few‑Shot Learners. NeurIPS 2020. https://arxiv.org/abs/2005.14165

2. Generalist AI. GEN‑1.5: Embodied Foundation Models are One‑Shot Learners. 2026. https://generalistai.com/blog/gen-1.5

3. Skild AI. Introducing S1: In-Context Learning for Robotics. 2026. https://www.skild.ai/blogs/s1

4. Duan, Y. et al. One-Shot Imitation Learning. NeurIPS 2017. https://arxiv.org/abs/1703.07326

5. Polanyi, M. The Tacit Dimension. University of Chicago Press, 1966.

6. Chan, S. C. Y. et al. Data Distributional Properties Drive Emergent In-Context Learning in Transformers. NeurIPS 2022. https://arxiv.org/abs/2205.05055

7. Raventós, A. et al. Pretraining Task Diversity and the Emergence of Non-Bayesian In-Context Learning for Regression. NeurIPS 2023. https://arxiv.org/abs/2306.15063

8. Wang, L. et al. Video Generation Models are General-Purpose Vision Learners. 2026. https://arxiv.org/abs/2607.09024

9. Silver, D. and Sutton, R. S. Welcome to the Era of Experience. 2025. PDF

© THE END 

For reprints, please contact this public account for permission

Submit or request coverage: liyazhou@jiqizhixin.com