RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination

Tencent Robotics X  ×  Futian Laboratory  ×  Tencent Hy Team

https://github.com/user-attachments/assets/e2359703-aa0a-4179-b34e-207b422bb50e

---

📌 Introduction

RxBrain (Hy-Embodied-RxBrain-1.0) is a unified multimodal foundation model for embodied cognition — one model that delivers three core capabilities:

• 🤖 Embodied Understanding & Reasoning — question answering and chain-of-thought over images and multi-frame video.
• 🔮 World State Prediction — imagine the near-future frames an action produces in the physical world.
• 🧩 Joint Subgoal Planning — decompose a task into steps, emitting for each step both the next action (language) and the goal image it should reach (vision).

---

📰 News

• [2026-07] 🚀 We release the fine-tuning code for VQA, Multi Frames Generation and Interleave Generation — see train/.
• [2026-07] 🎉 We release Hy-Embodied-RxBrain-1.0 — the technical report, inference code and model weights.

---

📋 TODO

• Open-source RxBrain-Bench, a comprehensive benchmark for embodied cognition.
• Open-source the fine-tuning code for Visual Question Answering (VQA), Multi Frames Generation, and Interleave Generation — see train/.
• Release the technical report, inference code, and model weights.

---

🚀 Quick Start

1. Environment

git clone https://github.com/Tencent-Hunyuan/Hy-Embodied-RxBrain-1.0.git
cd Hy-Embodied-RxBrain-1.0
pip install -r requirements.txt

Requirements: Python 3.10+, a CUDA GPU (for flash-attn).

Repository layout:

model/        UnifiedMoT model definition (3 streams: text / vision / generation)
vae_model/    FLUX VAE autoencoder loader
inference/    inference entry points  — T2I, multi-frame, VQA, interleaved planning
train/        fine-tuning pipeline    — see train/README.md
demo_cases/   bundled scenes for the interleaved-planning demo

model/ and `va