zju3dv
精选
INTACT-JEPA
INTACT is an end-to-end JEPA world model that learns an isomorphic intent-to-action interface, enabling search-free direct control for goal-conditioned robot tasks with high success rates after just one epoch of training.
AI 仓库情报摘要
FR-AI / ANALYSIS为什么值得关注
它通过可复现的解析管线（MANO到URDF无需优化）和明确的指标，填补了运动学手-物生成与物理合理性之间的鸿沟，能揭示生成抓取在物理仿真中是否真正有效，而不仅仅是生成漂亮的视频。
适合谁使用
- 开发手-物交互（HOI）生成模型的研究人员
- 计算机视觉领域评估抓取与操作合理性的学者
- 需要将基于MANO的交互序列放入仿真环境测试的机器人工程师
- 对比 MuJoCo 和 IsaacGym 在手-物交互任务中表现的仿真工程师
典型使用场景
- 用物理仿真检验HOI生成论文中的抓取效果
- 评估自定义手-物序列（如来自自己的模型）的抓取成功率
- 在同一交互上比较两个物理引擎（MuJoCo 和 IsaacGym）的行为差异
- 重放并评估来自HO-Cap等数据集的双手交互
项目优势
- 解析式MANO到URDF转换达到约0.02毫米关节精度，无需逆运动学或优化
- 一条命令即可实现运动学重放和完整物理仿真
- 支持自定义数据（单/双手捆绑格式）和两大主流仿真器
- 提供明确的轨迹跟踪指标（grasp_success），能揭示运动学重放掩盖的失败
使用前须知
- 需要注册并手动下载MANO和DexYCB数据集（未直接分发）
- IsaacGym后端需手动安装Preview 4（非必须但需额外步骤）
- 轨迹评估（grasp_success）仅限单手，双手会话仅渲染不评分
README 快速开始
DexHOI2Sim
Drop a hand-object interaction (MANO hand + object CAD) into a physics simulator and measure whether it holds up. Built to benchmark HOI-generation methods: take a generated grasp/manipulation, replicate it in MuJoCo and IsaacGym, and get back success-rate and physical-plausibility metrics — not just a pretty video.
The hand is the MANO2URDF 45-DOF hand: the real MANO mesh partitioned into 16 links, driven by joint angles computed analytically from the MANO pose (no retargeting optimization). Verified to reproduce MANO to ~0.02 mm (joints) / ~1.5 mm (vertices).
Example A — DexYCB 20200709-subject-01 / 20200709_142211 (003_cracker_box)
| MuJoCo — kinematic | MuJoCo — physics | IsaacGym — physics |
|---|---|---|
| box follows the hand (exact replay) | hand knocks the box — grasp fails | same, in IsaacGym |
Example B — DexYCB 20200709-subject-01 / 20200709_141754 (002_master_chef_can)
| MuJoCo — kinematic | MuJoCo — physics | IsaacGym — physics |
|---|---|---|
| hand grasps + lifts the can | can not held — grasp fails | same, in IsaacGym |
Kinematic replays the recorded motion exactly; physics reveals that this naive
replay does not actually hold the object (grasp_success: false, see below) — a
useful signal when benchmarking HOI-generation methods.
The hand is shaped by the MANO β (shape) parameter:
generate_urdf.pybuilds the URDF mesh from each subject'sbetas, so a different subject → a differently sized/shaped hand. The URDF is regenerated per β.
Install
conda env create -f environment.yml && conda activate dexhoi2sim
# or: pip install -r requirements.txt
# MANO models (register, non-commercial): put MANO_RIGHT.pkl / MANO_LEFT.pkl in
# mano2urdf/assets/ # https://mano.is.tue.mpg.de
# DexYCB (register): download to /root/data/dexycb # https://dex-ycb.github.io
IsaacGym is an optional backend (Preview 4, manual install). MuJoCo works out
of the box. Headless rendering uses EGL — set MUJOCO_GL=egl MUJOCO_EGL_DEVICE_ID=0
(the CLI does this for you).
Run
One command builds the hand URDF, computes per-frame joint angles, simulates, and evaluates:
python replicate.py \
--subject 20200709-subject-01 --session 20200709_142211 \
--backend both --mode physics --render --eval \
项目描述
Replicate MANO hand-object interactions in MuJoCo & IsaacGym and evaluate grasp success — a benchmark for HOI generation.
相关仓库与替代方案
根据分类、Topic 和编程语言匹配的相似项目。
lopopolo
精选
harness-engineering
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
slvDev
精选
esp32-ai
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.