zju3dv/INTACT-JEPA - 仓库详情

AI 仓库情报摘要

FR-AI / ANALYSIS

为什么值得关注

它消除了部署时传统潜在世界模型所需的大范围数值搜索，在四个视觉域上实现了零搜索直接控制，宏成功率达到 95.33%；共享编码器版本达到 89.39%，且无需单独的策略训练阶段或冻结编码器。

适合谁使用

机器人学习研究人员
世界模型从业者
强化学习工程师
控制理论学者

典型使用场景

无需动作搜索的定向机器人控制
基于共享编码器的多任务视觉控制
带可选局部优化的实时规划
从无奖励轨迹进行端到端世界模型训练

项目优势

零搜索直接控制在四个任务上仅用一个 epoch 即达到 95.33% 的宏成功率
受保护验证仅使用 CEM 300×30 的 23.44 分之一序列数，宏成功率高达 96.86%
共享编码器变体达到 89.39% 直接宏成功率，优于基于 CEM 的基线 (66.17%)
同构设计通过共享预测器耦合局部和目标意图，无需匹配端点或要求线性动力学

使用前须知

训练/评估代码和模型检查点尚未公开发布（计划未来发布）
仅在 LeWM 协议下的四个视觉控制域（PushT、Cube、Reacher、TwoRoom）上报告结果
该方法需要特定的 JEPA 骨干网络，对于极高维动作空间或非视觉观测可能需要适配

README 快速开始

INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models

Train a world model to answer the control query it will receive at deployment. End-to-end JEPA world modeling for goal-conditioned robot control without broad action search.

Junhan Sun1
&nbsp;&middot;&nbsp; Hao Zhao2,4,&dagger;
&nbsp;&middot;&nbsp; Guofeng Zhang1,3,&dagger;


1State Key Laboratory of CAD&amp;CG, Zhejiang University
2Institute for AI Industry Research (AIR), Tsinghua University
3InSpatio &nbsp;&middot;&nbsp; 4RoboParty Lab
&dagger;Corresponding authors














TL;DR &nbsp;|&nbsp;
Method &nbsp;|&nbsp;
Results &nbsp;|&nbsp;
Website &nbsp;|&nbsp;
Reproducibility &nbsp;|&nbsp;
中文

TL;DR

Forward latent world models learn what an action will do. At deployment, they usually recover the action for a desired change through broad numerical search. INTACT jointly learns the missing intent-to-action interface, so its conditional mean acts directly as a search-free controller while sampling is reserved for diversity or optional local verification. It needs no frozen encoder, extra policy-training stage, or globally linear latent dynamics.

1 epochend-to-end task-specific training
95.33%Direct four-task macro SR
0candidate sequences for Direct
2.9-5.5 msplanner-side Direct latency


96.86%Guarded A macro SR
384local verification sequences
23.44x fewerthan CEM 300x30
89.39%shared-encoder E5 Direct macro

Abstract

INTACT is an end-to-end JEPA that converts action-labelled, reward-free trajectories into a deployable intent-to-action interface. It is isomorphic in two explicit senses: the local and goal intent calls use an identical four-slot backbone-input graph with shared parameters, and both supported intent families acquire action-law semantics through that same predictor. Asymmetric gradients ground physical successors and keep future goals fixed as deployment anchors. The resulting representation preserves action-effective information from RGB observations while maintaining the world structure needed by forward prediction.

Method

Each action-labelled transition provides two state-conditioned intent families:

$$ m_t^{\mathrm{local}} = z_{t+1}-z_t, \qquad m_t^{\mathrm{goal}} = \mathrm{sg}(z_g)-z_t. $$

One

项目描述

INTACT: Isomorphic Intent-to-Action Learning for Search-Free World Models.

INTACT-JEPA