Tsinghua Team Builds HandsOn Dexterous Manipulation Benchmark, Evaluating Robots’ Dual‑Hand Capabilities Across 66 Tasks

Tsinghua team builds HandsOn dexterous manipulation benchmark, 66 tasks comprehensively evaluate robot dual‑hand capabilities

Edited by Chen Chen, Clio

The “exam paper” for robot manipulation needs to be rewritten for dexterous hands.

Previous robot manipulation benchmarks mostly focused on grippers. As first‑person (ego) human operation data continues to accumulate, how humans grasp, use tools, and perform fine contacts is being recorded on an ever larger scale. Dexterous hands have a structure similar to human hands and are expected to leverage this data for large‑scale learning, becoming a key direction in robot manipulation.

However, while dexterous hands keep evolving, the “exam paper” has not kept pace.

Dexterous hands have many active joints, tasks are contact-rich, often require bimanual collaboration, and must preserve earlier results over long action sequences; success is hard to judge, and pinpointing the step where failure occurs is also difficult. Benchmarks from the gripper era cannot directly answer these questions, and systems and reliable evaluation frameworks tailored to dexterous hands are still lacking.

Data and models are progressing, but a ruler to measure that progress is missing.

Recently, Ding Ning’s team at Tsinghua University, together with researchers from Beijing University of Posts and Telecommunications, Peking University, and Nature Will, built a composable simulation platform and evaluation benchmark called HandsOn for cross‑ontology dexterous manipulation, aiming to fill this gap.

HandsOn includes 66 human-validated tasks and 18,000 human-simulated teleoperation demonstrations, evaluating tool use, fine contact, bimanual collaboration, and task transfer. The platform supports 5 types of robotic arms, 6 types of dexterous hands, and about 200 rigid, articulated, and deformable object assets, which can be combined in randomized scenes.

HandsOn includes 18,000 curated simulated teleoperation demonstrations, with tasks divided into Core-50 and Transfer-16.

The benchmark’s report is far from easy: the team evaluated diffusion strategies, vision‑language action models, world action models, and other approaches. Results show that on the Core‑50 task, which provides training demonstrations, the best trained multitask strategy achieved a success rate of 29.1%. For Transfer‑16, which lacks target‑task training demonstrations, the average success rate of all eight trained multitask strategies was below 1%.

In addition, GPT‑6, without any benchmark data fine‑tuning, controlled the palm and fingers via the move_eef interface, achieving success rates of 5.2% on Core‑50 and 1.9% on Transfer‑16 across 660 trials.

The team also tracked stages such as grasping, contact, bimanual coordination, and goal‑state maintenance, finding that robots often complete the first half of an action but fail at critical moments.

Related papers, code, and data resources have been released.

• Paper arXiv: https://arxiv.org/abs/2610.26765

• Code GitHub: https://github.com/TsinghuaC3I/HandsOn

• Dataset Hugging Face: https://huggingface.co/datasets/Cyan666/HandsOn-Datasets

Why do dexterous hands need a new evaluation benchmark?

The challenge of dexterous hands is not just that they have more fingers than grippers; it also lies in the need to control continuously changing contact relationships during operation.

After picking up the mouse, the robot must avoid triggering a click prematurely; when setting two knobs, it needs to keep them simultaneously at the target positions; when placing a box lid, it must maintain the original alignment after releasing. These tasks all require the robot to coordinate finger motions, judge contact states, and ensure that completed actions are not disrupted by subsequent movements.

This also changes the way robot operations are evaluated. For grippers, whether the grasp is successful and whether the object is placed at the designated location are often the key criteria. For dexterous hands, merely holding an object is far from enough. The robot must actually use tools, perform fine manipulations, and maintain the correct object state throughout multi-step tasks. Progress on a single action does not mean the entire task has succeeded.

The accumulation of first‑person human operation data provides new opportunities for dexterous hands to learn these skills. However, even as demonstration data grows, two questions remain: Can the model reliably reproduce the learned operations? When faced with new objects, task combinations, or deformable items, can it transfer its existing skills?

Existing dexterous manipulation benchmarks already cover various robot bodies, functional operations, and bimanual collaboration. However, related research still has room for improvement in areas such as the scale of human demonstration data, unseen tasks reserved for evaluation, and diagnosis of failure processes.

HandsOn’s design starts from this point, separating the measurement of execution ability and generalization ability, and dividing the 66 tasks into two evaluation suites: Core-50 and Transfer-16.

Core-50 provides demonstration data for policy learning and evaluates policy performance through a new round of independent execution tests; Transfer-16 is dedicated to assessing generalization ability and does not include demonstration data of the target tasks in the training set.

HandsOn Suite: Core-50 includes demonstration content; Transfer-16 is for evaluation use only.

This design not only tests whether a robot can complete tasks with demonstrations, but also further examines whether existing skills can transfer to new operating conditions. Combined with staged diagnostics, researchers can also analyze whether failures occur during contact establishment, functional operation, bimanual coordination, goal state maintenance, or subsequent step transitions.

Core-50: After learning a task, can it be executed consistently?

Core-50 includes 50 tasks that provide training demonstrations, covering abilities such as functional understanding, numeric input, spatial relationships, joint manipulation, precise contact, and bimanual collaboration.

These categories correspond to specific everyday scenarios. When operating scissors, the robot needs to lift them off the table and complete three open‑close cycles while maintaining a grip; when setting a timer, it must input the specified minutes and seconds, then retract both hands after starting; unfolding a folding fan requires both hands to hold the outer ribs, spread it to the target angle, and hold it there. Making coffee further links steps such as picking up the cup, closing the door, receiving the coffee, and placing it, testing the coordination between actions.

Coffee preparation: multiple sub‑goals linked together

Take a cup from the cabinet, close the door, receive the coffee, then place it on the tray.

Bottle cleaning: sequence and two‑hand division of labor

One hand stabilizes the bottle while the other hand brushes the designated areas in order.

Open the folding fan: coordinated joint movement with both hands

Hold the outer fan ribs with both hands, open to the target angle and hold.

Set Timer: Numeric Targets and Button Operations

Enter the specified minutes and seconds, then start and retract both hands.

Suspended Open‑Close Scissors:Gripping and Functional Movements

Lift the scissors off the table, maintain the grip, and complete three open‑close cycles.

Trigger drill:Contact and function activation

Keep the drill bit in contact with the wood block while holding the trigger for half a second.

Six task instances in Core-50. Each task has a clear goal, action constraints, and completion criteria; the images are original simulation scene screenshots.

Task design expands along three dimensions.

• The first dimension is interaction type, including basic operations, functional use, and contact-intensive operations.

• The second dimension is task structure, covering single-stage operations, repeated operations, and long-duration tasks.

• The third dimension is object type, encompassing rigid bodies, objects with rotational or sliding joints, and deformable objects.

In Core-50, the three interaction types contain 17, 18, and 15 tasks respectively, and the three dimensions are cross‑combined, allowing the benchmark to cover operation scenarios of varying difficulty.

Transfer-16: Can it still succeed after changing objects and tasks?

Transfer-16 includes 16 evaluation tasks that do not provide demonstration training for the target tasks, further testing whether the model can transfer existing skills. Among them, 7 tasks examine asset or relation transfer, 6 tasks examine skill composition, and the remaining 3 involve deformable object manipulation.

Asset or relation transfer requires the model to face new objects or target relationships, identify the correct operation target, and complete the task. Skill composition requires the model to chain previously learned actions in the correct order. Deformable object manipulation introduces towels, sponges, and pillows, requiring the model to handle changes in object shape during the operation.

For example, in the task of handing over a block and then covering the box, the robot must first complete a two‑handed handover, then place the block into the box, and finally align and close the lid. Each sub‑action may have already appeared in other tasks, but the model still needs to determine whether the current step is finished and whether the result of the previous step can be maintained.

The requirements for continuously folding a towel are even more complex. The robot must perform two folds in succession and, after releasing, leave the towel stably on the table. Flexible objects change shape depending on the grasp location and force application, making the manipulation process harder to predict.

Mixer knob:Asset migration

Find the corresponding knob on the new device, adjust it to the target and hold.

Handover Block Rear Cover Box:Skill Combination

Complete the two‑hand handoff, place into the box and release, then align the box cover and close it.

Continuous folding towel:Flexible object manipulation

Fold twice consecutively, then release to let the towel stay stably on the desk.

Compressed Pillow Packing: Deformation and Two-Handed Handling

Squeeze, lift, and place into the cardboard box; keep stable after release.

The representative task in Transfer-16, demonstrating asset migration, skill composition, and deformable object manipulation.

It can be seen that Core-50 mainly tests learning and execution abilities on demonstrated tasks, while Transfer-16 further evaluates whether existing abilities can be transferred and combined.

Ensuring failures are accurately recorded

To ensure trustworthy evaluation, HandsOn defines a task card for each task, specifying the goal, standard initial state, expected final state, allowed and prohibited actions, and the evidence for determining task completion.

Completion conditions can be judged based on object pose, joint state, or image information. The research team also performed manual verification of the tasks and established an independent review process to check whether the success labels match the actual physical state.

In multi-step tasks, the evaluation also records the sequential progress of each stage. For example, a block handover task distinguishes when both hands are in contact with the object and when the receiving hand can support the object independently. A skill-combination task checks whether prerequisite steps have been completed and whether previous results are maintained.

This allows researchers to differentiate between two scenarios: the model did not complete the task at all, or it completed part of it but failed in subsequent steps.

After 18,000 demonstrations, what did the robot learn?

Unlike gripper data, demonstrations for dexterous hands need to record fine-grained finger-level motions. HandsOn uses human teleoperation to convert human hand actions into demonstrations for the dexterous hand, which also aligns with the development direction of first‑person human operation data.

Around this task suite, HandsOn collected about 56 hours of human operation recordings, involving at least 10 operators and more than 1,050 collection sessions. Operators wore MANUS data gloves and used a wrist‑tracking device to control a simulated robot, while the system synchronously saved images, states, actions, and language information.

Of the 18,000 demonstrations, 15,178 are used for training and 2,822 for validation. Each interaction includes front‑view and left/right wrist‑view three‑camera streams: the front view shows the tabletop layout and hand positions, while the wrist views provide close‑up observations of the fingers and objects. For tasks requiring precise contact, the close‑up footage supplements information about how the fingers approach, grasp, and manipulate the objects.

How does the success rate change as the data scale increases? The research team selected the tasks “put the ball into the bowl” and “lift the scissors and open/close them”, using π0.5 to conduct single-task experiments, gradually increasing the training demonstrations to 10, 50, 100, 200, and finally using the full training pool.

π0.5 single-task demonstration scale experiment, with full training pools of 438 and 439 demonstrations respectively. Each checkpoint is trained for 20,000 updates and tested for 100 episodes; the shaded area represents the 95% Wilson interval. A single training seed is used, and the interval does not include uncertainty from the training seed or subset selection.

The results show that the success rate of the ball‑placement task increased from 7% to 83%, and the scissors task from 1% to 44%, with both tasks continuously improving across the five test scales. The full training pools contain 438 and 439 demonstrations respectively, each checkpoint was trained for 20,000 updates, and performance was measured over 100 new closed‑loop execution rounds.

These results provide an intuitive reference for data benefits. As demonstrations increase, the model achieves higher success rates on both tasks; however, a substantial gap remains between the scissors operation and the ball‑placement task, highlighting the difficulty of combining stable grasping with tool‑use actions. The experiment used a single training seed, and the reported 95% Wilson interval reflects uncertainty across test rounds; variations due to training seeds and subset selection still need further validation.

The data can also be extended visually. HandsOn supports high‑fidelity re‑rendering of the same segment of operation, allowing changes to tabletop material, lighting, room background, and slight camera offsets while preserving the original state and action supervision. Researchers can thus generate different observations from a single interaction to examine how strategies adapt to visual variations.

In the robot configuration, HandsOn establishes a simulation correspondence around the real ROKAE robotic arm and the first‑generation Wuji Hand, with each hand containing 20 active joints. Kinematics, joint limits, asset scaling, camera calibration, as well as state and action coordinates are all recorded correspondingly. The main camera and the wrist camera work together, allowing the overall layout and hand details to enter the same data and evaluation pipeline.

Simulation of the toy airplane scenario compared with the real‑world view, with the wrist image enlarged.

The platform supports 5 types of robotic arms, 6 types of dexterous hands, and about 200 assets. This data collection and quantitative evaluation uniformly use the ROKAE—Wuji and PhysX configurations. The team also provides real‑robot execution videos that demonstrate the simulation adaptation strategy; under conditions without target‑task real‑robot demos, they qualitatively showcase contact, manipulation, and pose maintenance. For real‑world performance, these videos can be consulted, though the evidence scope differs from the quantitative evaluation in simulation.

29.1% success rate – where does dual‑hand coordination become difficult?

HandsOn evaluated 10 methods, including 5 visual‑language action models (VLA), 3 world action models (WAM), a task‑specific Diffusion Policy, and zero‑shot control reference GPT‑6. The first 9 methods were trained on HandsOn demonstrations, with 8 of them using multitask training.

On Core-50, nine training methods collectively completed nearly 45,000 valid tests, with the highest average task success rate for a multitask strategy achieved by OpenWAM at 29.1%. The differences between tasks are quite pronounced: grasping, transporting, and some joint operations are relatively easy, while using tools on a third object or performing precise assembly remains difficult.

The results of transfer evaluation are even harsher. On Transfer-16, the average task success rates of eight multitask training strategies are all below 1%, and the single-task Diffusion Policy did not participate in this evaluation. When faced with new assets, skill combinations, and deformable objects, the ability to translate existing training experience into complete task success remains limited.

When complete success is rare, stage progress provides an entry point for further analysis. The dual‑knob task is an example: each hand operates a knob, moving both sides to the target state while satisfying hold and release conditions. In 800 attempts across 8 multitask models, 219 times each side reached the target, 38 times they arrived simultaneously, and finally only 5 times met the full conditions.

The gap mainly lies in how two local results can co‑occur. A common situation is that the hand that arrives first continues turning, and before the other hand finishes, the already achieved result changes.

The block handoff reflects the temporal requirements between the two hands. In 367 rounds where both hands simultaneously held the block, 188 times the receiving hand took over independently; in the remaining 179 rounds, 175 of them subsequently dropped. The timing of when a hand releases must be coordinated with the supporting state of the receiving hand.

A total of 800 rounds with two knobs; among the 367 shared-contact rounds of the block handover, 188 achieved independent takeover by the receiving hand.

Longer tasks also added sequencing requirements. In a combined task that required covering the kettle first and then the box, out of 400 attempts, 340 moved the box lid first. These records provide traceable questions for future research: how the model determines that the current step is completed, how it preserves existing results, and when it initiates the next operation.

The study also examined whether actions change with linguistic targets by keeping the visual scene constant and swapping the target object in the instruction. Some checkpoints still show limited responses to object names, indicating that target selection also requires continuous verification. A single task failure may involve contact control, or it may occur earlier at the stage of target comprehension and action selection.

Zero‑shot testing of GPT‑6 provides another set of observations. In each round it reads three RGB images, robot state, task instruction, and execution feedback, and specifies palm pose with move_eef along with up to 40 finger joint targets.

Without any HandsOn demonstration training or fine‑tuning, the model was tested 10 times on each of the 66 tasks, achieving 26 successes on Core‑50 and 3 successes on Transfer‑16, corresponding to success rates of 5.2% and 1.9%.

The action logs show that the model can propose plans related to the premise of the operation. For example, when folding a towel, it suggested first making a small test lift, confirming that the towel edge moves freely, and then continuing to fold. This round ultimately failed, reflecting the difficulty of translating plans into contact and motion control.

For robots heading toward everyday environments, these details are directly relevant to real-world use. As task configurations, demonstration data, and evaluation methods become publicly available, researchers can test new approaches under shared settings and analyze the sources of success-rate changes throughout the execution process. This benchmark provides a set of sustainable experimental conditions for studying how a robot’s two hands gradually master daily operations.

Conclusion: Ensuring Capability Advances Stand Up to Real-World Tests

Hands are among the most delicate organs of humans. Through their hands we use tools, reshape environments, and accumulate experience about objects and the world with every operation.

For robots to enter human living and working spaces, they also need to integrate perception, judgment, and action into continuous physical interaction. In the robotics industry, the maturity of dual‑hand capabilities will determine how much actual work a general model can undertake.

The robotics industry is gradually entering a stage that requires the accumulation of shared evidence. Technical demos show what capabilities are possible, while large‑scale deployment demands that these capabilities withstand repeated execution, varying scenarios, and long‑term operation.

In this process, public benchmarks serve to bridge research results and application needs. As model capabilities improve, evaluations must also incorporate new tasks and constraints from real‑world applications, so that score gains correspond to broader and more reliable operational abilities.

The long‑term value of HandsOn will gradually become evident as researchers participate: new methods drive performance improvements, failure cases raise new research questions, and application needs push continuous refinement of testing standards. For the industry, such accumulation helps define clearer capability boundaries, providing a more solid basis for R&D investment and implementation expectations.

© THE END 

For reprints, please contact this public account for permission

Submit or request coverage: liyazhou@jiqizhixin.com