Why it is worth attention

It provides a flexible framework integrating state-of-the-art video generation models (Wan2.2, Cosmos-Predict2) with multiple action modeling paradigms, includes pretrained checkpoints for LIBERO and RoboTwin 2.0, and achieves high performance (89.5% on RoboTwin 2.0).

Who it is for

Robot learning researchers
Embodied AI practitioners
Video generation researchers exploring action prediction
Developers building robot policies with world models

Use cases

Training and evaluating WAMs on LIBERO and RoboTwin benchmarks
Comparing different WAM families (MoT, shared-DiT, feature-conditioned)
Extending with new backbone adapters or action model variants
Reproducing research results with provided pretrained checkpoints

Strengths

Modular design with typed recipe system (YAML to dataclasses) separating backbones and WAM families
Supports multiple WAM families (mot_wam, shared_dit_wam, feature_conditioned_action_model) with different attention mechanisms
Includes pretrained checkpoints for two major benchmarks (LIBERO, RoboTwin 2.0) with documented rollout commands
Integrates with popular video generation backbones (Wan2.2, Cosmos-Predict2) and provides clear adapter interfaces

Considerations

Early research release with limited documentation and ongoing roadmap (e.g., missing technical report)
Currently only supports four WAM families and two backbone adapters; additional benchmarks and variants are planned
No formal citation or BibTeX entry yet; reliance on external pretrained models may increase deployment complexity

README quick start

StarWAM: A Generalizable Codebase for World-Action Models

StarWAM is a research codebase for building World-Action Models (WAMs): robot policies that combine generative video/world models with action prediction modules. It is designed for modular experimentation with world-model backbones, action representations, and training recipes.

This repository is an early research release. More WAM variants, benchmarks, model checkpoints, and technical details will be added.

News

2026/07: RoboTwin 2.0 integration — dual-arm MoT recipe, client/server rollout adapter for the official RoboTwin harness (89.5% avg over 50 tasks).
2026/07: Initial StarWAM codebase prepared with Wan2.2 and Cosmos-Predict2 backbone adapters, LIBERO training/rollout recipes, MoT WAM, Shared-DiT WAM, and feature-conditioned WAM support.

Highlights

World-model backbones: reuse pretrained video generation models as robot world models, e.g., Wan2.2 and Cosmos-Predict2.
Multiple WAM families:
- mot_wam: multi-stream video/action experts with mixed attention, e.g., Motus / FastWAM-style world-action modeling.
- shared_dit_wam: shared-DiT/register-token video-action prediction, e.g., DreamZero / LingBot-VA-style shared-token formulations.
- feature_conditioned_action_model: action prediction conditioned on video/world-model features, e.g., Video-IDM, Mimic-Video / World2Action, and StarVLA-WM4A-style variants.
Benchmark recipes: benchmark-specific data loading, normalization, text embedding caches, training recipes, and rollout utilities.
Typed recipe system: YAML recipes are loaded into Python dataclasses without requiring Hydra.

Repository Layout

starwam/                     # Core Python package
  backbone/                  # Wan2.2 / Cosmos-Predict2 backbone adapters
  wam/                       # WAM wrappers and taxonomy-level model families
  modules/                   # DiT blocks, MoT, scheduler, shared-DiT modules
  action_model/              # Action expert builders
  data/                      # LeRobot dataset and text-cache utilities
  training/                  # Trainer, losses, flow utilities, entrypoint
  tools/                     # Preprocessing utilities
  utils/                     # Checkpoint and config helpers
configs/                     # Accelerate / DeepSpeed configs
examples/

Description

A Generalizable Codebase for World-Action Models

Related repositories

Similar projects matched by category, topics, and programming language.

lopopolo

Featured

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI & Machine LearningAI Agents

2,390

slvDev

Featured

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI & Machine LearningLarge Language Models

1,960

littledivy

Featured

mimic

mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.

AI & Machine Learning

1,482