From Hardware and Data to Models: How Sharpa Systematically Advances Dexterous Manipulation

From Hardware and Data to Models: How Sharpa Systematically Advances Dexterous Manipulation

Teleoperation data is expensive, slow, and hard to scale, so more and more teams are turning their attention to human video. Projects such as Nvidia’s EgoScale pre‑train robot policies on 20,854 hours of first‑person human video and fit a logarithmic‑linear curve; work like HumanScale goes further, arguing that with the same amount of data, first‑person video as pre‑training data can outperform real‑world teleoperation data.

Scaling behavior pre‑trained on human data, arXiv:2602.16710

For gripper grasping, the logic of this trajectory is relatively smooth. However, when the actuator is replaced by a dexterous hand with more than twenty degrees of freedom, the problem becomes complex, because the video records how the hand moves but does not capture how much force is applied. For tasks that require fine manipulation such as unscrewing bottle caps, plugging connectors, or slicing apples, success or failure hinges precisely on the millimeter-scale motion that occurs after contact.

Sharpa Wave Dexterous Hand

The data challenge of dexterous manipulation is essentially a contact problem.

Around this issue, Sharpa, a robot company focused on dexterous manipulation, has defined three main exploration directions:

• Co-design of hardware and software will shift data collection from robot‑centric to human‑centric

• Decouple “motion” from “interaction,” delegating contact to high‑frequency, tactile low‑level control

• Use tactile feedback as an anchor to learn universal manipulation representations across robot bodies

Along these major directions, Sharpa’s hardware platform has become a key carrier for many cutting‑edge studies. In recent months, teams from Google, NVIDIA, and others have conducted a series of embodied‑intelligence experiments and frontier explorations using Sharpa hardware. For example, Google’s latest Gemini 4 Argon, released a few days ago, showcased a robot sand‑bag‑throwing experiment that used Sharpa’s dexterous hand.

At the same time, Sharpa has partnered with teams from the University of Hong Kong, UC Berkeley, Tsinghua University, Shanghai Jiao Tong University, and others to jointly advance data collection, model learning, and cross-ontology generalization research in the field of dexterous manipulation.

Machine Heart recently spoke with Sharpa’s academic lead Zhang Kaifeng to try to clarify what questions these efforts are addressing and how they fit together into a complete picture.

One Apple Stumped VLA

First, look at a concrete task: peeling an apple with two dexterous hands.

In this task, one hand must hold the apple and keep rotating it, while the other hand controls the peeler, advancing along the apple’s surface. The curvature of the apple changes, the skin thickness varies, and the gripping force and peeling angle must be adjusted accordingly. The two hands do not perform symmetric actions; one hand is responsible for “stability” and “rotation,” the other for “peeling,” and they must coordinate their rhythm.

Sequence of two‑hand apple peeling in the MoDE‑VLA paper

In the MoDE-VLA paper co-authored by Sharpa with Shanghai Jiao Tong University, Shanghai AI Laboratory and other institutions,  MoDE-VLA, the researchers used the mainstream VLA model π0 as a baseline for this task. The result was a 0% success rate, with only 8% peeling completion.

• Paper title: Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA

• Paper URL: https://arxiv.org/pdf/2603.08122

This is not surprising. The paper opens by noting that VLA models are already quite mature for grasp-and-place tasks using parallel grippers, but they fall short on dexterous tasks that involve rich contact, bimanual coordination, and in-hand manipulation. Where is the problem? Is the model not large enough, the data insufficient, or something else?

To answer this question, we first need to clarify exactly what distinguishes a dexterous hand from a gripper.

According to Zhang Kaifeng, the difference between the two is not just the number of degrees of freedom. He says: “Traditional robotic arms and grippers simplify all manipulation problems into issues that can be solved kinematically, whereas dexterous hands involve dynamics.” The gripper’s action space is basically just open and close; as long as you move it to the right position and close it at the right moment, the task is largely completed. A dexterous hand, however, must invoke various low‑level atomic skills such as grasp, pull, twist, pinch (motion primitive), and Zhang estimates that the human hand has on the order of dozens of such skills.

This means that dexterous manipulation cannot rely solely on “aiming at the position and then executing.” The magnitude of force, contact state, and slip of the object in the hand all continuously affect the next action.

At the hardware level, high‑DOF, tactile‑enabled dexterous hands have become increasingly mature. Zhang Kaifeng believes that the biggest bottleneck after the hardware is data, “especially data for human‑like dexterous manipulation.” The difficulty of dexterous manipulation data lies precisely in the need to include contact information.

Two kinds of data, a dilemma

Currently, dexterous manipulation data mainly comes from two sources.

The first type is teleoperation: a human remotely controls a robot to complete tasks via visual devices or exoskeletons, recording the robot’s own motions. In recent years, the academic community has introduced a series of teleoperation systems such as AnyTeleop, Open‑TeleVision, Bunny‑VisionPro, and HOMIE, continuously improving immersion and usability. This kind of data corresponds directly to the robot body and can be used for training.

Dual‑hand apple‑cutting teleoperation demo from the MoDE‑VLA paper

However, teleoperation has an unavoidable issue: the operator cannot feel the robot’s contact. “Operators find it hard to obtain high‑fidelity sensations of what the robot feels during operation,” says Kai‑Feng Zhang, because the forces and tactile feedback at the robot’s fingertips are difficult to relay to the operator in real time. In fine manipulation, this means the operator must guess forces visually, which hampers efficiency and data quality.

The second approach is human‑centric data collection: people directly perform tasks with their own hands, recording them with motion‑capture devices, data gloves, or head‑mounted cameras. Work such as DexCap, DexUMI, DEXOP, OSMO, etc., falls into this category. The human hand provides full tactile feedback, the operation feels natural, and the data is easier to scale up.

The issue is that a human hand and a robot hand are not the same entity. Zhang Kaifeng breaks the gap down into two layers.

The first layer is visual: the image shows a human hand, not a robotic hand.

The second layer is about motion, and it is even more fundamental. “Whether you use the most advanced motion‑capture devices, a motion‑capture studio, or any other method, what you actually obtain is hand state, not motion,” he explained, “What’s missing between motion and state? The signals of force and tactile feedback.” Driving a dexterous hand’s motors requires torque and joint‑angle signals, which are not directly present in human hand data.

Thus, researchers face a dilemma: the more the data‑collection method mirrors natural human motion and feels comfortable for the operator, the larger the embodiment gap with the robot; the more the method aligns with the robot’s structure, the higher the demands on the operator and the poorer the experience.

Around the long‑term goal of “Towards Human‑Like Dexterous Manipulation”, the Sharpa academic team has established three main lines of work to address this dilemma:

• Resolve the ontological gap between human‑centered data and teleoperation data, answering “how data become robot capabilities”;

• Decouple “motion” from “interaction”, reconstruct contact control through simulation and atomic skills; 

• Learn invariant representations across ontologies, answering “how models transfer between different robots and sensors”.

According to Kaifeng Zhang, the three main lines “are parallel and mutually reinforcing”.

Main Line One: Bridge the “embodiment gap” and turn human data into robot action fuel

To break out of the dilemma “the more comfortable the collection, the harder the data to use,” we must first answer: can we preserve the naturalness and efficiency of human‑centric collection while also capturing the missing force and tactile information? A straightforward idea is to record tactile data at the collection side, which requires ontology‑free data acquisition.

To achieve this, we need ontology‑free data acquisition.

The term “ontology‑free” means that the robot does not need to be present during data collection; a person wearing sensor devices operates directly. The key to this approach is to retain the naturalness and tactile feedback of manual operation while ensuring that the collected data can ultimately be used by robots.

UVTA (Unified Visual‑Tactile‑Action Modeling / 统一视觉 – 触觉 – 动作建模) is a representative work in this field, completed by Shanghai Jiao Tong University, Beijing General Artificial Intelligence Research Institute, Sharpa, and Beijing Institute of Technology. According to Kaifeng Zhang, the research team added fingertip tactile sensors to the Sharpa AE01 exoskeleton data glove: a 22‑degree‑of‑freedom passive linkage records finger motion, each fingertip’s force‑sensitive resistor array records touch, and a wrist‑mounted camera records a first‑person view.

• Paper title: Unified Visual‑Tactile‑Action Modeling from Human Demonstrations for Dexterous Manipulation 

Paper URL: https://arxiv.org/pdf/2609.34182

After collecting the data, how do we prove it is useful? UVTA’s approach is to combine human demonstrations with a small amount of teleoperated robot data for joint training, mapping both into a shared visual‑tactile‑action space, allowing the model to simultaneously predict future actions and tactile signals. Human data serve only as training supervision; during deployment the robot always executes its own actions. As Zhang Kaifeng puts it, it is “jointly modeling the relationship between vision and touch to predict future states and tactile signals, and finally grounding them into action signals.”

UVTA overview. VTPM converts visual‑haptic input into 20‑dimensional piezoresistive tactile unit tokens, while the Transformer extracts 384‑dimensional visual tokens. These two tokens are concatenated into a unified 404‑dimensional interaction representation and fed to the diffusion action head and tactile regression head. The action loss and tactile loss supervise their respective prediction heads, and their joint objective optimizes the shared Transformer to learn a unified tactile‑perceptual visual representation.

On five contact‑rich tasks—page turning, screwing a light bulb, flipping a switch, ball classification, and liquid transfer—UVTA achieves an average success rate of 70%, compared to 29% for the baseline T‑Rex. More notably, a set of scaling experiments fixed 150 robot demonstrations per task and gradually increased human demonstrations from 0 to 1,000. The average success rate across three tasks rose from 0% to 84%, and the paper states that no saturation was observed even at 1,000 demonstrations.

Scaling curve of the number of human demonstrations and success rate in the UVTA paper

Another work  DexEMG attempts to further lower the data collection barrier. It infers hand pose from forearm surface EMG signals and then remaps it in real time to the 22‑DOF Sharpa Wave dexterous hand, replacing bulky exoskeletons and visual motion capture. Because it does not rely on cameras, it can still be operated remotely even when fingers are occluded by objects.

• Paper title: DexEMG: Towards Dexterous Teleoperation System via EMG2Pose Generalization 

Paper URL: https://arxiv.org/pdf/2603.05861

However, in the real world there is another, even larger class of assets: tens of thousands of hours of human monocular video on the Internet that contain no tactile information at all. How can we bridge the human‑machine action gap when the video does not record touch?

In the Dex‑X solution, a collaboration between Sharpa and Xu Mengdi’s team at Tsinghua University, researchers forged a different path: they precisely extract the 3D spatial motion trajectories of hands and objects from monocular RGB human videos, then feed these trajectories into a physics simulation environment. The simulation engine, using collision bodies and contact dynamics, retroactively supplies the invisible hand‑object interaction forces that are absent in the real video, thereby achieving effective transfer from pure human video to real‑world tactile policies.

• Paper title: Learning Visual‑Tactile Dexterous Manipulation From Human Videos with Simulated Interaction

• Paper URL: https://arxiv.org/pdf/2609.07747

It can be seen that, from UVTA’s real‑device tactile augmentation, to DexEMG’s lowering of collection barriers, to Dex‑X’s simulated dynamics completion, thread one is shifting the focus of data collection from “forcing humans to adapt to machines” to “enabling the system to understand human touch”. The significance of this line of work lies in turning the qualitative question “whether human data can be used for dexterous manipulation” into a quantitative one that can be answered with scaling curves.

But the video never has force. The motion can be redirected, yet contact cannot be generated out of thin air. How to fill this gap is the question the second main thread must answer.

Main Thread Two: Decouple “Movement” and “Interaction,” Rebuild a General Foundation for Contact Control

Even with the data in hand, the difficulty of dexterous manipulation still centers on the moment of contact. Zhang Kaifeng noted that the hardest part of teleoperation is that the robot’s sensations cannot be transmitted to the operator with high fidelity when it contacts an object.

Since contact is the bottleneck, a natural idea is to handle movement and interaction separately: the movement part can be learned from human videos or even internet videos; the interaction part, i.e., the contact‑rich segment, can be handed over to simulation and reinforcement learning.

This split has a physical rationale. The large‑scale motion before contact is more of a kinematics problem, and human video can provide rich priors; the compression, sliding, and force control after contact are dynamics problems, highly sensitive to errors, which is precisely the information most lacking in video.

This idea is reflected in Sharpa’s  CraftNet hierarchical model unveiled at CES 2026.

Illustration of CraftNet’s three‑layer architecture

CraftNet divides the robot’s decision‑making into three layers:

• System 2 is the “reasoning brain,” handled by a vision‑language model that interprets commands and decomposes tasks, running at about 1 Hz;

• System 1 is the “motor brain,” responsible for motion planning and coarse pre‑contact control, around 10 Hz;

• System 0 is the “interaction brain,” which continuously adjusts the fingers during contact using real‑time tactile feedback, about 100 Hz.

After breaking it down, new questions arise: What interface connects motion and interaction? What information must the higher level pass to the lower level, and how should the lower level respond?

Around this question, Sharpa and the collaborating team explored several different answers.

The first interface is a pedal. This goes back to the apple at the beginning. MoDE-VLA paper, the researchers first used reinforcement learning to train a set of atomic hand‑operation skills, called IMCopilot, including maintaining a stable grasp, rotating an object around a specified axis, and so on. When collecting data, the operator used an exoskeleton to control arm movements; when the apple needed to be turned, they pressed the pedal with their foot, and the hand rotation was handed over to IMCopilot to complete automatically. Zhang Kaifeng described this as “using both hands and feet”: the foot presses the pedal to drive the low‑level apple rotation, while the hand focuses on peeling.

System overview. (a) Teleoperation system, including exoskeleton, VR headset, and foot pedal. (b) VR view, integrating the robot camera feed with real‑time force and haptic feedback overlay. (c) Robot platform for performing rich contact tasks.

The results are quite good. In the hand‑held operation test, the success rate with IMCopilot assistance was 89%, compared to only 34% for direct teleoperation. High‑quality apple‑peeling data were thus collected. In the autonomous execution stage, the VLA model can also invoke the underlying IMCopilot skill, and together with a dedicated expert pathway for processing force and haptic signals, forms MoDE‑VLA. Across four tasks—apple peeling, test‑tube organization, gear assembly, and charger plugging—the average success rate of MoDE‑VLA increased from a baseline of 15% to 34%, apple‑peeling success rose from 0% to 30%, and peeling completeness improved from 8% to 73%.

Zhang Kaifeng believes this is only the first step. “In the future, as long as I provide some target points and some operation intents, I can drive the underlying controller of the dexterous hand to perform relatively sophisticated human‑like actions,” he said. “Leave these contact‑rich operations to the low‑level controller, and the upper layer only needs to supply intents and coarse trajectory motions.”

The second interface is point clouds. A typical example is Dex‑X, which follows a top‑down approach. It reconstructs the hand‑object interaction process from monocular human videos, then uses a reinforcement‑learning policy in simulation to track this reference motion. The missing contact‑force information in the video is filled in by the physics engine in simulation; the paper calls the simulation a “tactile completion engine”. Finally, the policy in simulation is distilled into a student policy that takes point clouds, proprioception, and fingertip tactile data as input, deployed on the real Sharpa Wave dexterous hand, achieving a 93% success rate on the real‑world block‑grasping task.

The training framework of DEX-X. After transferring human demonstrations to a simulated environment, it uses reinforcement learning to train a privileged-state expert, incorporating demonstration references, object states, and tactile contact information. This expert is then distilled into a multitask visual‑tactile policy that operates on a unified contact point‑cloud representation, embedding fingertip tactile feedback into scene geometry and fusing vision, force, and proprioception for policy learning. The resulting policy can zero‑shot transfer to a variety of real‑world dexterous manipulation tasks.

The same top‑down approach also appears in other teams’ research based on Sharpa hardware. The Do as I Do project by Jitendra Malik’s Berkeley group and Pieter Abbeel’s team reconstructs hand‑object interactions from everyday human videos and redirects them into dexterous hand motions. On 655 wild‑video reconstruction results, the redirection success rate increased from 25% in previous methods to 71%, and they completed 10 real‑robot tasks on Sharpa Wave, including stirring, pouring, wiping, and more.

Institutions such as UC Berkeley, Sharpa Robotics, and the University of Hong Kong proposed GALATEA, which extends the idea to generative video. It directly leverages diffusion video models to generate hand‑object interaction processes based on the first‑frame image and natural language instructions, and optimizes them through binocular depth anchoring and contact perception, reconstructing precise 3D hand and object motion trajectories from pixels. Facing the inevitable physical hallucinations and noise in generated videos, GALATEA builds a tracking‑based reinforcement learning pipeline that combines contact‑ and lift‑aware rewards, using an adaptive policy optimization algorithm (SAPG) to force visual imagination in simulation to “align” with physically stable control strategies.

In experiments, the system filtered about 2,000 high‑quality trajectories from 2,500 generated videos and successfully deployed over 1,500 strategies in simulation, ultimately distilling a single controller that, without any retraining, achieves closed‑loop generalized execution of diverse tasks such as grasping, non‑grasp pushing/pulling, and aerial pose fine‑tuning.

• Paper title: Grounding Generated Video Plans in Simulation Towards Versatile Dexterous Controllers

• Paper link: https://arxiv.org/pdf/2609.10050

The third interface is the latent space. Zhang Kaifeng believes that a top‑down approach may not be the final solution. Another path is bottom‑up: train a large number of atomic skills in simulation, then distill them into a unified framework. This framework is System 0.

TacBPM is the concrete implementation of this idea, and all authors are from Sharpa. Researchers first train expert policies for in‑hand reorientation on eight spheres of different sizes, then distill them into a tactile‑conditioned latent‑space controller. Downstream tasks do not need to learn from scratch; they simply freeze this prior and learn residual actions in the latent space. In a multi‑task setting with complex objects, PPO trained on raw actions achieves only a 1% success rate, whereas TacBPM reaches 70%; in a simulated grasping task with a robotic arm and dexterous hand, success rates for some objects improve from less than 1% when trained from scratch to 97%. The paper shows that this prior can also adapt quickly to new object shapes, new tasks, and new embodiments.

• Paper title: TacBPM: A Tactile‑conditioned Behavior Prior Model for Dexterous Reorientation

• Paper URL: https://arxiv.org/abs/2609.18174

The Stanford team of Jeannette Bohg and C. Karen Liu has two works based on Sharp Wave that echo this direction:

• SimToolReal  trains an object-centric simulation policy that, without retraining for each tool, can perform zero‑shot real‑world operations for 6 tool types and 24 tasks, improving performance by 37% over redirection and fixed‑grasp methods and was selected for an oral presentation at RSS 2026.

• Play2Perfect  lets a dexterous hand first undergo task‑agnostic “free‑play” pre‑training in simulation, then fine‑tune for precise assembly; it achieves 33× higher sample efficiency than training from scratch and reaches a 60% success rate on tight‑fit insertion tasks with a 0.5 mm clearance.

Some demos based on Sharpa Wave released by the SimToolReal team

Zhang Kaifeng categorizes the Malik team’s work as top‑down, and the work of Bohg and Liu’s team as bottom‑up; the two complement Sharpa’s own explorations on the same roadmap.

Pedal, point cloud, and latent space—these three interfaces correspond to different division of labor between the upper and lower layers. But their direction is consistent: strip the contact layer out of the upper‑level model and hand it to a high‑frequency, haptic lower layer.

Main Thread Three: Anchored by Haptics

The first two main threads focus on where the data comes from, while the third thread concerns how the data is used.

Current robot policies are often tied to specific robots, dexterous hands, or even tactile sensors. When the hardware changes, the data must be recollected and the model retrained. Zhang Kaifeng explains the necessity of cross‑embodiment learning from two perspectives:

• From the model’s standpoint, a good robot base model should learn robust representations. “Cross‑embodiment data can help our model learn a better representation—one that is operation‑oriented and independent of the embodiment.” Only then can the model quickly adapt during downstream fine‑tuning for different embodiments.

• From an application perspective, downstream robots come in many forms, and true generalization must address different embodiments.

So, what kind of information is best suited to carry this ontology‑independent representation? Sharpa’s answer is touch.

Zhang Kaifeng uses a coffee cup as an example. A human hand, Sharpa’s dexterous hand, and a gripper are three different ontologies, but when picking up the same cup, some information remains unchanged. “For example, the amount of force I use to pick up this cup should be independent of whether I’m using a human hand or a robotic hand,” he says, and this type of information primarily comes from touch. The ontology can change, but the operational process implied by tactile signals stays the same, making touch a “anchor” for cross‑ontology learning.

FTP‑1 provides the most direct validation of this claim. This work, conducted jointly by Tsinghua University, Sharpa, Shanghai Jiao Tong University, Berkeley, ETH Zurich, Fudan University and others, proposes a universal tactile foundation strategy for cross‑tactile‑sensor applications.

• Paper title: FTP-1: A Generalist Foundation Tactile Policy Across Tactile Sensors for Contact-Rich Manipulation

• Paper link: https://arxiv.org/abs/2606.13102

The challenge lies in the heterogeneity of tactile sensors: some output images, some output arrays, and some output force and state. FTP-1 maps these different forms of input into a shared latent space, then processes them with a unified Transformer, completing pre‑training on about 3000 hours of data from 21 types of tactile sensors.

The results show that on sensor configurations seen during pre‑training, FTP‑1 yields a 17.2% improvement; on two unseen sensor configurations, the success rate increases by 31%. To confirm that the gain indeed comes from tactile information, the researchers also trained a control model NTP‑1 without tactile data, and FTP‑1 outperforms it on the transfer tasks. This demonstrates that large‑scale tactile pre‑training helps the model learn reusable representations across ontologies and sensors.

There is another approach to handling ontology differences along this main line. Kai‑Feng Zhang groups NVIDIA’s EgoScale and Berkeley’s T‑Rex under this line: they do not explicitly address the gap between human hands and robot hands, but instead directly use first‑person human data and then perform mid‑training with paired human‑robot data, allowing the model to “digest” the differences itself and learn invariant representations.

EgoScale was completed through a collaboration between NVIDIA, Berkeley, and the University of Maryland, with authors including Jim Fan, Zhu Yuke, Xu Danfei, and others. The researchers pre‑trained on 20,854 hours of first‑person human video, then performed intermediate alignment training with about 50 hours of human data and 4 hours of robot data. On the 22‑degree‑of‑freedom Sharpa Wave, the average success rate improved by 54% compared to a baseline without pre‑training. An important reason for choosing Sharpa Wave is that it has a 1:1 size with a human hand and a high‑DOF structure, making it easier to redirect human motions to robot motions.

8× speed video

T‑Rex introduces tactile sensing into this framework. This work was carried out by institutions such as Berkeley, NVIDIA, and Stanford, with authors including Fei‑Fei Li, Jim Fan, Jitendra Malik, Pieter Abbeel, and others. The researchers built a 100‑hour tactile‑synchronized teleoperation dataset based on Sharpa Wave, covering 22 motion primitives and more than 200 everyday objects, and designed a variable‑frequency hybrid Transformer expert architecture: the motion expert generates actions at about 5 Hz, while the tactile expert updates in real time at about 20 Hz based on tactile signals. Across 12 contact‑rich tasks such as peeling, wiping, squeezing, inserting, and extracting, T‑Rex achieves an average success rate of 65%, roughly 30 percentage points higher than the strongest baseline, EgoScale.

When these works are viewed together, a clear research relay emerges. EgoScale demonstrated the scaling effect of human data on Sharpa Wave; T‑Rex introduced tactile sensing on the same platform, using EgoScale as a baseline to raise the average success rate by about 30 percentage points; UVTA, which involves Sharpa, then used T‑Rex as a comparative baseline to further advance the utilization of human tactile data. Only when different teams build on the same hardware platform can research results be compared and accumulated layer by layer.

Of course, whether tactile feedback is necessary is not without debate in academia. Some studies argue that vision alone is sufficient to infer contact—for example, FELT attempts to generate fingertip tactile signals directly from RGB images and indeed improves the performance of purely visual strategies. In response, Kaifeng Zhang offered two points.

• First, the “degenerate solution” problem: a light touch and a hard press look almost identical on the screen—“for example, if I press this table, you can’t really tell how much force I’m using.”

• Second, occlusion: when the finger wraps around an object, a large amount of visual information is lost, and tactile feedback often becomes one of the few usable signals.

The two approaches are not necessarily mutually exclusive. Generating tactile feedback from vision can reduce data collection costs, while genuine tactile signals may determine how fine-grained the contact phase can be.

Sharpa’s understanding of haptics has also evolved.

Zhang Kaifeng recalls that the early team focused on “modal competition” in multimodal learning: if a task could be mostly solved with vision, the model would easily take shortcuts, relying on vision and neglecting touch. The SaTA project, done in collaboration with Tsinghua, was work from this stage; it anchored tactile features to the hand kinematics coordinate system, allowing the policy to infer the geometric position of objects in the hand. In sub‑millimeter‑precision tasks such as blind USB‑C insertion and light‑bulb installation, success rates improved by up to 30 percentage points. Later, the team gradually realized that touch is more like a low‑level “reflex arc” and should reside in System 0 for high‑frequency adjustments and adaptation.

The collaboration with NVIDIA, Tacmap, fills this gap on the simulation side. It unifies simulated and real tactile signals into the same geometric representation, enabling hand‑in‑hand rotation policies trained only in simulation to transfer to real robots with zero samples.

• Paper title: Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map

• Paper link: https://arxiv.org/pdf/2602.21625

Three lines converge at the base

Looking back, the three main threads ultimately point to the same spot.

In the architecture described by Zhang Kaifeng, tactile and proprioceptive perception are placed in the low‑level System 0, while modalities that carry higher‑level semantics such as vision, language, and even sound reside in the upper layers, System 1 and System 2. The first two main threads address where contact data comes from, and the third main thread addresses how contact information can be reused across embodiments; both are supplying fuel to System 0. “Currently we are working around data,” Zhang Kaifeng said, “in the future these pathways will gradually merge together.”

The logic of merging corresponds to Sharpa’s “flywheel” concept: good hardware yields good data, good data trains good models, and real‑world deployment then feeds back to drive iteration of hardware, data, and models.

The hardware serves as the research platform. Zhang Kaifeng reduces the design of Sharpa Wave to “biomimicry”: same structure as a human hand, 1:1 scale, 22 degrees of freedom.The previously mentioned EgoScale and Do as I Do both take these points as prerequisite conditions for research. From the researcher’s perspective, he also notes two points: first, reliability—“you can’t say it breaks after an hour, and then it takes two weeks to fix it”; second, good calibration and consistency, so the gap between simulation and the real device is small, allowing researchers to smoothly perform reinforcement learning in simulation and then transfer it to the real hardware.

At the end of September this year at IROS 2026, Sharpa released the D01 full system, the W02 dexterous hand, and the AE01 haptic feedback data glove. Among them, the AE01 is a 22‑degree‑of‑freedom exoskeleton glove that provides vibrotactile feedback at the fingertips, targeting precisely the teleoperation and data‑collection stages.

In real deployments, Sharpa and Dairy Queen jointly opened the “Robot Blizzard” restaurant on Wujiang Road in Shanghai on August 29. The robot must independently complete a 55‑step process from ordering to delivery. Reports indicate that 98% of these steps involve tactile interaction. Such long‑duration, continuous real‑world scenarios are exactly the source of “real‑world feedback” in the flywheel.

The Last Millimeter

How far is dexterous manipulation from being truly usable? Zhang Kaifeng’s assessment is relatively optimistic. He believes that single-rigid-body humanoid dexterous manipulation could see breakthroughs within the next 3 to 6 months; overall, “these problems could be solved on a roughly two‑year timescale.”

Behind this judgment lies an even larger variable. The spatial reasoning ability of general large models is improving rapidly, and problems once thought to require robot data are now being tackled by models trained without any robot data. Will System 2 and System 1 eventually be absorbed by large‑model companies? Zhang Kaifeng does not shy away from this question. His view is that if you get System 0 and the model itself right, you can quickly adapt to this new trend.

This may also be a perspective for understanding Sharpa’s academic main line. The higher-level reasoning and planning may increasingly be handled by general-purpose models; but the millimeter after the finger contacts an object requires high-frequency tactile feedback, tightly coupled control with the body, and specially collected and designed data. This layer may be the part of the hierarchical architecture of embodied intelligence that is hardest for general models to replace.

In the past year, more and more teams have begun embedding tactile sensing into model architectures, delegating contact to high-frequency low-level components; this approach is shifting from a single company’s viewpoint to a shared topic in academia. Over a dozen papers co-authored by Sharpa with NVIDIA, Hong Kong University, Berkeley, Tsinghua, Shanghai Jiao Tong University and others constitute a fairly concentrated set of examples of this shift. The “last millimeter” of dexterous manipulation is being advanced together by them.

© THE END

For reprints, please contact this public account for permission

Submit articles or request coverage: liyazhou@jiqizhixin.com