zhuang2002 GitHub avatar

Self_Gradient_Forcing

zhuang2002

Self Gradient Forcing (SGF) 通过有界的两遍回放机制恢复自生成因果记忆中缺失的上下文梯度路径,使仅用5秒窗口训练的视频扩散模型能够生成分钟级的长视频,同时保持更强的身份、布局和时间稳定性。

Stars

106

7 天增长

暂无数据

Fork 数

2

开放 Issue

1

开源协议

Apache-2.0

最近更新

2026-07-23

AI 仓库情报摘要
FR-AI / ANALYSIS

为什么值得关注

它直接解决了当前视频生成中长视频外推这一关键瓶颈,无需更长的训练序列即可实现原生长视频外推,并提供了基于 Wan 视频模型生态系统的开源代码、预训练模型以及清晰的推理和训练指南。

适合谁使用

  • 从事长视频合成研究的视频生成科研人员
  • 使用扩散模型进行视频生成的AI从业者
  • 构建自动化长视频内容创作工具的开发者
  • 研究生成模型时间一致性的学者

典型使用场景

  • 从短文本提示生成延长的视频序列
  • 将短视频片段外推为完整的分钟级场景以用于原型制作
  • 生产具有稳定物体身份的连贯长动画
  • 作为长视频生成研究的基线或模块

项目优势

  • 实现原生长视频外推(约240秒,16fps),而模型仅用5秒窗口训练
  • 提供两种可配置变体(逐帧和逐块),并附带即用型推理和训练脚本
  • 完全开源:包含论文、预训练检查点(1.3B和14B两种规模)和完整代码
  • 支持单节点和多节点分布式训练,节点自动注册

使用前须知

  • 需要特定的环境配置(Causal-Forcing、Wan 基础模型、Flash Attention),且推理推荐8块GPU,计算资源需求高
  • KV缓存几何参数需通过 shell 变量手动设置,可能需针对不同硬件或输出长度进行调整
  • 除展示效果外,缺乏失败案例或定量性能基准的详细说明

README 快速开始

🌀 Self Gradient Forcing

Native Long-Video Extrapolation

     

Junhao Zhuang, Shiyi Zhang, Yuxuan Bian, Yaowei Li, Yawen Luo, Yijun Liu, Weiyang Jin, Songchun Zhang, Xianglong He, Xuying Zhang, Haoran Li, Haoyang Huang, Zeyue Xue, Nan Duan

Joy Future Academy, JD

⭐ If Self Gradient Forcing is useful for your research, please consider starring this repository.

🔥 News

  • 2026-07-23: Paper, model checkpoints, inference scripts, and training code are publicly released.

🧠 Method Overview

Self Gradient Forcing (SGF) recovers the missing context-gradient path for self-generated causal memory through a bounded two-pass replay, enabling models trained with only a 5-second window to extrapolate to minute-scale videos with stronger identity, layout, and temporal stability.

🛠️ Installation

The environment follows the Causal-Forcing setup.

conda create -n self_gradient_forcing python=3.10 -y
conda activate self_gradient_forcing

pip install -r requirements.txt
pip install flash-attn --no-build-isolation
python setup.py develop

⬇️ Download Weights

bash scripts/download_weights.sh

The script uses the Hugging Face CLI command hf by default. Set HF_CLI=huggingface-cli if your environment still uses the older command name.

It downloads:

  • Wan base models to wan_models/Wan2.1-T2V-1.3B and wan_models/Wan2.1-T2V-14B.
  • All Causal-Forcing initialization checkpoints under checkpoints/init/framewise/ and checkpoints/init/chunkwise/: ar_diffusion.pt, causal_cd.pt, and causal_ode.pt.
  • Released SGF inference checkpoints to checkpoints/framewise/ar/model.pt and checkpoints/chunkwise/ar/model.pt.
  • The training prompt list to prompts/vidprom_filtered_extended.txt.

🚀 Inference

The default prompt file is prompts/test_prompt.txt with 8 prompts. The launcher uses 8 GPUs when at least 8 GPUs are visible; otherwise it falls back to single-GPU serial inference. By default it generates 963 latent frames, which decode to about 240 seconds of video at 16 fps. The inference script takes the release setting name (framewise or chunkwise) and selects the matching config and checkpoint automatically:

  • framewise config: configs/self_gradient_forcing_framewise.yaml
  • chunkwise config: `configs/self_gradie

项目描述

Self Gradient Forcing (SGF) recovers the missing context-gradient path for self-generated causal memory through a bounded two-pass replay, enabling models trained with only a 5-second window to extrapolate to minute-scale videos with stronger identity, layout, and temporal stability.

相关仓库与替代方案

根据分类、Topic 和编程语言匹配的相似项目。

lopopolo
精选
lopopolo GitHub avatar

harness-engineering

Harness Engineering is a methodology for improving coding agent outputs by carefully crafting the environment around them—providing curated context, tools, and executable constraints that encode an organization’s nonfunctional requirements and cumulative lessons.

AI 与机器学习AI 智能体
2,390
slvDev
精选
slvDev GitHub avatar

esp32-ai

A 28.9 million parameter language model runs on an $8 ESP32-S3 microcontroller entirely on-device, generating simple stories at about 9.5 tokens per second.

AI 与机器学习大语言模型
1,960
littledivy
精选
littledivy GitHub avatar

mimic

mimic captures traffic from any iOS or web app and automatically generates a Python client library that lets you call the app's API like a regular library.

AI 与机器学习
1,482