zju3dv
Featured
INTACT-JEPA
INTACT is an end-to-end JEPA world model that learns an isomorphic intent-to-action interface, enabling search-free direct control for goal-conditioned robot tasks with high success rates after just one epoch of training.
DexHOI2Sim
DexHOI2Sim drops a hand-object interaction (MANO hand + object CAD) into MuJoCo and IsaacGym physics simulators and measures whether the grasp physically holds up, providing success-rate and trajectory-error metrics.
Stars
12
7-day growth
No data
Forks
0
Open issues
0
License
No data
Last updated
2026-07-16
AI repository intelligence
FR-AI / ANALYSISWhy it is worth attention
It bridges the gap between kinematic hand-object generation and physical plausibility by offering a reproducible, analytic pipeline (MANO-to-URDF without optimization) and a clear metric that reveals whether a generated grasp actually works in physics, not just in videos.
Who it is for
- Researchers developing hand-object interaction (HOI) generation models
- Computer vision scientists benchmarking grasp and manipulation plausibility
- Roboticists who need to test MANO-based interaction sequences in simulation
- Simulation engineers evaluating MuJoCo vs. IsaacGym for HOI tasks
Use cases
- Benchmarking generated grasps from HOI-generation papers against physical reality
- Evaluating custom hand-object sequences (e.g., from your own model) for success rate
- Comparing physics behavior across two simulators (MuJoCo and IsaacGym) on the same interaction
- Replaying and assessing bimanual interactions from datasets like HO-Cap
Strengths
- Analytic MANO-to-URDF conversion achieves ~0.02 mm joint accuracy with no retargeting optimization
- Provides both kinematic replay and full physics simulation with a single command
- Supports custom data (single-hand or bimanual bundles) and two major simulators
- Includes a concrete trajectory-tracking metric (grasp_success) that reveals failures hidden by kinematic replay
Considerations
- Requires registration and manual download of MANO and DexYCB datasets (not redistributed)
- IsaacGym backend is optional but requires manual installation of Preview 4
- Trajectory evaluation (grasp_success) is single-hand only; bimanual sessions render but skip scoring
README quick start
DexHOI2Sim
Drop a hand-object interaction (MANO hand + object CAD) into a physics simulator and measure whether it holds up. Built to benchmark HOI-generation methods: take a generated grasp/manipulation, replicate it in MuJoCo and IsaacGym, and get back success-rate and physical-plausibility metrics — not just a pretty video.
The hand is the MANO2URDF 45-DOF hand: the real MANO mesh partitioned into 16 links, driven by joint angles computed analytically from the MANO pose (no retargeting optimization). Verified to reproduce MANO to ~0.02 mm (joints) / ~1.5 mm (vertices).
Example A — DexYCB 20200709-subject-01 / 20200709_142211 (003_cracker_box)
| MuJoCo — kinematic | MuJoCo — physics | IsaacGym — physics |
|---|---|---|
| box follows the hand (exact replay) | hand knocks the box — grasp fails | same, in IsaacGym |
Example B — DexYCB 20200709-subject-01 / 20200709_141754 (002_master_chef_can)
| MuJoCo — kinematic | MuJoCo — physics | IsaacGym — physics |
|---|---|---|
| hand grasps + lifts the can | can not held — grasp fails | same, in IsaacGym |
Kinematic replays the recorded motion exactly; physics reveals that this naive
replay does not actually hold the object (grasp_success: false, see below) — a
useful signal when benchmarking HOI-generation methods.
The hand is shaped by the MANO β (shape) parameter:
generate_urdf.pybuilds the URDF mesh from each subject'sbetas, so a different subject → a differently sized/shaped hand. The URDF is regenerated per β.
Install
conda env create -f environment.yml && conda activate dexhoi2sim
# or: pip install -r requirements.txt
# MANO models (register, non-commercial): put MANO_RIGHT.pkl / MANO_LEFT.pkl in
# mano2urdf/assets/ # https://mano.is.tue.mpg.de
# DexYCB (register): download to /root/data/dexycb # https://dex-ycb.github.io
IsaacGym is an optional backend (Preview 4, manual install). MuJoCo works out
of the box. Headless rendering uses EGL — set MUJOCO_GL=egl MUJOCO_EGL_DEVICE_ID=0
(the CLI does this for you).
Run
One command builds the hand URDF, computes per-frame joint angles, simulates, and evaluates:
python replicate.py \
--subject 20200709-subject-01 --session 20200709_142211 \
--backend both --mode physics --render --eval \
Description
Replicate MANO hand-object interactions in MuJoCo & IsaacGym and evaluate grasp success — a benchmark for HOI generation.
Related repositories
Similar projects matched by category, topics, and programming language.
lopopolo
Featured
harness-engineering
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
slvDev
Featured
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.