Primitive-Informed Sampling-Based MPC for Multi-Fingered Dexterous Manipulation

Anonymous submission

Low-dimensional manipulation primitives bias the sampling distribution of a sampling-based MPC, while joint-level residuals adapt the motion to the current hand–object configuration. On a physical Allegro hand this yields 100% success (86/86) on continuous in-hand rotation without palm support, where unstructured joint-space sampling never completes a single revolution, even with 6× more samples.

Abstract

We present a primitive-informed sampling-based model predictive control (MPC) framework for multi-fingered dexterous manipulation. Sampling-based MPC evaluates candidate control trajectories through forward simulation without requiring gradients through complex contact dynamics. However, directly sampling these trajectories in the high-dimensional joint space of a dexterous hand is inefficient and makes performance strongly dependent on the sampling distribution.

Our framework biases sampling using low-dimensional manipulation primitives that encode coordinated finger motions, while simultaneously optimizing joint-level residuals to adapt these motions to the current hand–object configuration. Task-related rollout constraints reject infeasible trajectories during forward simulation, improving the effective use of the sampling budget.

We evaluate the approach on a 16 DoF Allegro hand using a synchronized MuJoCo digital twin. Ablations show that both the primitive and residual are necessary for reliable continuous in-hand rotation, that increasing the sampling budget alone does not recover this coordination, and that rollout constraints substantially improve success rate. A primitive extracted for one object size transfers to other sizes and remains effective under model mismatch. The framework further supports grasping, object reorientation, and coordinated arm–hand manipulation, using primitives extracted from both a simulation-trained policy and human hand-motion data.

Method

System overview. Low-dimensional manipulation primitives bias trajectory sampling toward coordinated motions, while joint-level residuals adapt the sampled actions to the current hand–object configuration. Candidate trajectories are evaluated through forward simulation, and the optimized mean action sequence is executed on the physical system.

A MuJoCo digital twin of the arm, hand, and object is synchronized with the physical system at about 100 Hz. At each update, the cross-entropy method samples candidate spline-parameterized trajectories, rolls them out in MuJoCo, and refits the sampling distribution to the lowest-cost elites; the leading portion of the result is executed while the next update is computed.

Instead of sampling every joint independently, hand commands are reconstructed as u = q̄ + Φz + δq: a few coordinated motion primitives Φ, extracted by PCA from a simulation-trained policy or human hand-motion data, are mixed by coefficients z, and a joint-level residual δq adapts the motion to the current hand–object configuration. Task constraints checked at every simulated step terminate infeasible rollouts early.

Real-World Results

All experiments run on a physical 16-DoF Allegro hand mounted on a Franka arm, with object pose from motion capture. Controller settings are listed under Implementation Details below.

Experimental tasks. a) In-hand rotation, b) Reorientation, c) Grasping, d) Coordinated arm–hand manipulation.

In-hand rotation: No primitive vs. Ours

The object is held without palm support, so any update that fails to find a grasp-maintaining motion drops it. Direct joint-space sampling stalls at the rotation reachable with the initial grasp, while Ours rotates continuously through finger gaiting without a single drop.

No primitive (joint-space sampling)

Ours (primitive + residual + constraint)

More samples do not replace the primitive

Without the primitive, a six-fold larger sampling budget still never completes a rotation, because independently sampled joint motions rarely produce the coordinated regrasp that finger gaiting requires. Beyond a moderate budget, extra samples even hurt Ours: slower replanning lets sim-to-real discrepancies accumulate.

In-hand rotation performance versus sampling budget N. Top: rotation speed of successful rotations (left axis) and success ratio (right axis) for Ours (circles) and No primitive (squares). Bottom: maximum accumulated rotation per attempt for No primitive. Mean MPC update time is shown below each N. Error bars denote ± SD.

Ablations: primitive, residual, and constraint are all needed

Removing either the primitive or the residual eliminates all successful rotations: the primitive supplies the coordinated structure and the residual the local flexibility to adapt it. Removing the rollout constraint roughly halves the success ratio without changing the speed of the rotations that do succeed, so constraints improve reliability, not speed.

In-hand rotation ablations. Successful-turn rotation speeds are shown for Ours and No constraint, with each point representing one full rotation. Success ratios are shown for these two conditions and the pooled No primitive and No residual ablations. For the first two conditions, each success-ratio data point is computed over one three-minute trial.

One primitive transfers across object sizes and model mismatch

The primitive extracted from the 40 mm object keeps rotating 35 mm and 45 mm objects with nearly unchanged success and no re-extraction, and the controller also persists under ±50% errors in the modeled friction and mass, where large errors only slow rotation or increase drops.

40 mm (primitive source)

35 mm

45 mm

A human-motion primitive enables grasping

With grasping primitives extracted from human hand-motion data, the hand lifts the object from an open configuration in nearly every trial. Without them, sampled motions push or tilt the object instead of closing the fingers in coordination, and no grasp succeeds.

No primitive

Ours (human-motion primitive)

Object reorientation: constraints reduce drops

Tracking a cyclic heading command needs no finger gaiting, which isolates the effect of the rollout constraint. With it, most trials complete the full sequence; without it, most end in an early drop, while tracking accuracy while the object is held is the same in both cases.

No constraint

With constraint

Coordinated arm–hand reach–grasp–transport

Adding the desired arm pose to the latent action lets the same framework reach, grasp, and transport the object to a target pose in most trials. This sequence is never programmed; it emerges from online optimization of the task objective, and in the right video the robot even rotates the object non-prehensilely before grasping it.

Reach, grasp, and transport

Non-prehensile rotation, then grasp and transport

Implementation Details

Settings shared across experiments.

Parameter	Value
Planning horizon	0.5 s, K = 6 spline knots
CEM elites N_e	10 per update
Rollout budget N	350 (hand-only tasks), 500 (arm–hand task)
MPC update time	~38 ms at N = 350 (AMD Ryzen Threadripper PRO 7995WX, 192 threads)
State update / command rate	~100 Hz digital-twin sync, 333 Hz commands to Allegro
Residual std floors	σ^explore = 1.0 rad, σ^refine = 0.10 rad
Primitive-coefficient std floors	σ^explore = p^max − p^min, σ^refine = 0.3 (p^max − p^min)
Explained-variance threshold γ	90% (4 rotation primitives, 5 grasping primitives)
Rotation primitive source	SAC policy trained in simulation on the 40 mm AF hexagon, no domain randomization (5 trajectories, ~1 min)
Grasping primitive source	NinaPro: 25 right-handed subjects, 20 grasp motions
Rollout constraint	Object–table contact terminates the rollout with a large failure cost
Rotation success	One full revolution without a drop; ψ^des kept one revolution ahead
Grasp success	Object lifted ≥ 5 cm from the table
Arm–hand success	Object within 3 cm of the desired position
Statistics	Two-sided exact Wilcoxon rank-sum permutation tests; two-sided Fisher exact tests for binary outcomes