为什么值得关注

它克服了扩散模型在音乐到舞蹈生成中传统的 20 秒时长限制，实现了分钟级的连贯视频，具有高时间稳定性，并支持五种不同的舞蹈类型。

适合谁使用

专注于视频生成和扩散模型的 AI 研究人员
舞蹈内容创作者和音乐视频制作人
将 AI 生成的舞蹈集成到应用中的多媒体开发者
研究生成模型中长时时间一致性的学术研究人员

典型使用场景

为任意音乐曲目自动生成编排好的舞蹈视频
为现场表演或游戏动画制作舞蹈序列原型
为音乐流媒体平台生成背景舞蹈内容
通过不同舞蹈风格的视觉演示辅助舞蹈教育者

项目优势

实现了分钟级（超过 60 秒）高分辨率舞蹈视频生成，超越了以往扩散模型的限制
采用分层框架（全局关键帧 + 局部细化）实现长程时间一致性
支持五种舞蹈类型（中国古典舞、K-Pop、街舞、踢踏舞、拉丁舞），并具备音频和文本条件控制
引入基于光流的损失函数和动态帧率自适应，维持运动连续性和细节

使用前须知

推理需要大量计算资源（测试环境为 8 × NVIDIA A800 80GB GPU）
两阶段流程（全局再到局部）增加了复杂性和处理时间
依赖特定软件版本（CUDA 12.4、PyTorch 2.6 等），可能限制可移植性

README 快速开始

💃 Wan-Dancer

💜 Project &nbsp&nbsp ｜ &nbsp&nbsp 🖥️ GitHub &nbsp&nbsp | &nbsp&nbsp🤖 MS Space&nbsp&nbsp | &nbsp&nbsp🤖 MS Model&nbsp&nbsp | &nbsp&nbsp🤗 HF Model&nbsp&nbsp | &nbsp&nbsp 📑 Paper &nbsp&nbsp

Generating long-duration, high-quality, rhythmic dance videos from music with global structure and temporal continuity

📄 Abstract

Generating long-duration, high-definition, and rhythmically synchronized dance videos directly from music remains a significant challenge, primarily due to the temporal constraints of current diffusion models, which typically fail beyond 20 seconds. Existing approaches, whether they rely on intermediate 3D skeletons or on end-to-end video synthesis, suffer from temporal drift, identity inconsistency, and repetitive motion patterns when extended to longer horizons. To address these limitations, we propose a novel hierarchical framework for minute-scale coherent music-to-dance generation. Our method decouples the process into global keyframe planning and local temporal refinement, leveraging full-track musical context to ensure long-range coherence. Key innovations include dynamic frame rate adaptation via time-mapped RoPE embeddings for precise alignment, an optical-flow-based loss function to enhance motion continuity, and motion-speed control to preserve high-fidelity details during rapid movements. Extensive experiments demonstrate that our framework surpasses the conventional duration barrier, generating stable, 720p/30fps videos exceeding one minute with superior temporal stability. Furthermore, the model exhibits robust versatility across five distinct dance genres, conditioned on both audio and textual prompts, establishing a new state-of-the-art in coherent, long-form dance video synthesis.

⚙️ Environment Setup

We tested the code on:

OS: Ubuntu 22.04
Hardware: 8 × NVIDIA A800 80GB GPUs
Python: 3.10.14

✅ Install Dependencies

cd /path/to/Wan-Dancer
python -m venv venv_wan_dancer
source venv_wan_dancer/bin/activate

# Install package in editable mode
pip install -e .

# Install additional and specific versions dependencies
pip install moviepy loguru librosa
pip install https://mirrors.aliyun.com/pytorch-wheels/cu124/torch-2.6.0+cu124-cp310-cp310-linux_x86_64.whl
pip install torchvision==0.21.0
pip install diffusers==0.34.0
pip install yunchang==0.5.0

Wan-Dancer