🎬 VideoCoCo: Code as CoT for Physics-Faithful Video Generation

Official repository for VideoCoCo, a physics-faithful video generation pipeline that uses code as a chain-of-thought to draft physics before committing to pixels.

[🤗 Weights] [💻 Code]

💥 News

• [2026.07.29] We release the Agent Skills, a toy dataset, and the inference code. Tuned weights are uploading to the 🤗 Hub.

🪄 Draft Before Generation

We propose VideoCoCo, an interleaved reasoning paradigm that carries a physical prior through an explicit visual draft before committing to pixels.

Our method 🎨 first has a code agent write simulation code and render it in a sandbox as a neutral white/clay proxy video that carries the correct motion, causality, and physics — meaning is expressed by shape, transparency, deformation, and coverage, never by color.

Then we 🔎 verify the proxy against the physical plan (a caused state must stay hidden until its causing transition), and 🖼️ restyle the proxy into a photorealistic video driven by a per-case edit instruction.

📦 What's in this repo

• skill/ — the five Agent Skills forming the pipeline: physical-state-planner → physical-video-blender-implementer → blender-mcp-video → seedance-edit-prompt → seedance-distill.
• data/toy_cases/ — 8 hand-checked video-to-video (v2v) triplets.
• inference/ — batch inference scripts + a patch against upstream OmniWeaving.
• 🤗 mickyhimself/VideoCoCo — the tuned transformer.

🎬 Toy dataset

data/toy_cases/ — 8 v2v triplets, one directory per case:

data/toy_cases/
├── manifest.jsonl                 # one JSON line per case (index)
├── 0000_buoyancy/
│   ├── video.mp4                  # source: neutral white/clay physics proxy
│   ├── seedance.mp4               # target: photoreal restyle
│   └── edit_prompt.txt            # instruction used to restyle proxy -> photoreal
└── ...

Each manifest.jsonl line:

{"case_id": "0000_buoyancy", "source": "0000_buoyancy/video.mp4", "target": "0000_buoyancy/seedance.mp4", "instruction": "...", "category": "buoyancy"}

• source (video.mp4) — a grayscale/white-material rend