Open Dreamer is an open-source, real-time JAX/Flax implementation of the Dreamer 4 world model pipeline, providing training code for a causal video tokenizer and action-conditioned latent dynamics model on Minecraft/VPT-style gameplay.
302
No data
24
1
NOASSERTION
2026-07-26
This repository offers a complete, production-ready training pipeline for a state-of-the-art world model (Dreamer 4) with an impressive live browser demo that lets you play a generated Minecraft world in real time, making advanced world modeling accessible to practitioners.
An open, real-time implementation of the Dreamer 4 world-model pipeline in JAX/Flax.
🌐 Website & Blog Post · 🎮 Live Demo · ⚡ Inference Code
Authors
Diego Marti Monso* · Francesco Sacco* · Edward Hu
REAL-TIME DEMO POWERED BY
Open Dreamer is a simple, performant, and easy-to-use JAX/Flax NNX implementation of the Dreamer 4 world model — trained on Minecraft/VPT-style gameplay and playable in real time. This repository holds the training pipeline: a causal video tokenizer, an action-conditioned latent dynamics model, and the tools to generate rollouts and measure quality.
This repo currently supports:
The fastest way to experience Open Dreamer is the live, in-browser demo — step into a generated Minecraft world and play it in real time, with a Game ⟷ Dream toggle that hands the stream from the real game to the world model frame by frame. No setup required; the real-time model runs in the cloud on Reactor.
The website also walks through how the model works, with interactive figures and result galleries.
To roll out the trained model locally on your own video and actions — generating new frames from an MP4 and a matching action sequence — use the inference repository:
The rest of this README covers training the model from scratch.
uvpip install uv
uv sync
source .venv/bin/activate
The dependency lock targets CUDA 12 JAX. If your machine needs a different JAX build, in
Open-source Dreamer world-model implementation in JAX
Similar projects matched by category, topics, and programming language.
Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.
A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.
mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.