🐱 meituan-longcat  ·  Open Source  ·  MIT License

Audio-Driven Avatar
Video Generation

LongCat-Video-Avatar 1.5 delivers production-ready, commercial-grade avatar synthesis with Whisper-Large lip sync, 8-step distilled inference, and robust long-video generation.

🤗 Download Model GitHub Repo 🌐 Project Page
MIT Model Tasks Inference Audio
508
Benchmark Pairs
770
Human Evaluators
6
Application Scenarios
8
Inference Steps (NFE)
⚡ Capabilities

Key Features

🎙️

Whisper-Large Audio Encoder

Replaces Wav2Vec2 with Whisper-Large-v3, yielding significantly smoother and more natural lip dynamics with precise phoneme alignment.

⚡

8-Step DMD2 Distillation

Advanced step distillation accelerates inference to just 8 NFE, balancing cost-effective serving with exceptional visual fidelity.

🎭

Stylized Generalization

Robustly generalizes beyond realistic humans to anime characters, animals, and complex real-world conditions like object handling.

👥

Multi-Person Support

Dual-audio modes: Merge mode (parallel streams) or Concatenation mode (sequential speech). Full multi-character video synthesis.

📹

480P & 720P Resolution

Compatible with both 480P and 720P output, controlled via the --resolution flag. Production-ready super resolution.

🔄

Long-Video Continuation

Multi-segment generation via --num_segments. Maintains full identity consistency and temporal stability across segments.

💾

INT8 Quantization

Use --use_int8 to load the INT8 quantized DiT model for significantly reduced VRAM usage (v1.5 only).

🔒

Identity Consistency

Strict identity preservation across frames and segments. Accurate lip-synchronization and full-body temporal stability in long generations.

🎯 Supported Tasks

What You Can Generate

Three native generation modes, all running on the same model checkpoint.

🔊
--stage_1=at2v

Audio-Text-to-Video

Generate an avatar video from an audio clip + text prompt alone. No reference image needed.

🖼️
--stage_1=ai2v

Audio-Image-to-Video

Animate a reference portrait image with your audio clip. Preserves the exact identity from the image.

🎬
--num_segments=N

Video Continuation

Generate long-form videos by chaining multiple segments. Maintains temporal coherence and identity.

👥
multi_audio_to_video.py

Multi-Person Animation

Animate multiple characters simultaneously with independent or merged audio streams.

📊 Human Evaluation

Benchmark Results

508 image-audio pairs across 6 scenarios, 2 languages, 2 visual styles. 770 crowdsourced evaluators.

Expert Quality Assessment (4 Dimensions)

Physical Rationality4.6 / 5
Audio-Visual Harmony4.5 / 5
Temporal Stability4.7 / 5
Identity Consistency4.6 / 5

Subjective Human-Likeness (vs Commercial Models)

LongCat-Avatar 1.5 ⭐4.4 / 5
Commercial Model A4.1 / 5
Commercial Model B3.8 / 5
Commercial Model C3.6 / 5
💡 Usage Tips

Get the Best Results

01

Optimal Audio CFG (3–5)

Audio CFG works best between 3–5. Increase the value for stronger lip synchronization accuracy. Default is 4.

02

Rich, Descriptive Prompts

Longer prompts yield better results. Include character appearance, actions, clothing, and scene context — e.g. "A young woman with long black hair is speaking and smiling, wearing a white blouse, sitting in a bright café."

03

Reference Image Index

Set --ref_img_index between 0–24 for best identity consistency. Set to 30 to reduce repeated actions in longer generations.

04

Always Use --use_distill for v1.5

The --use_distill flag is required when using --model_type avatar-v1.5. It enables 8-step DMD2 distillation.

05

Save VRAM with INT8

Add --use_int8 to load the quantized DiT model, significantly reducing VRAM usage without major quality loss (v1.5 only).

🛠️ Quick Start

Run Locally

Get up and running in a few commands. Requires a GPU with ≥16GB VRAM (INT8 mode) or ≥24GB VRAM (full).

# Clone the repository git clone --single-branch --branch main https://github.com/meituan-longcat/LongCat-Video cd LongCat-Video # Create conda environment conda create -n longcat-video python=3.10 conda activate longcat-video # Install PyTorch (adjust CUDA version) pip install torch==2.6.0+cu124 torchvision==0.21.0+cu124 torchaudio==2.6.0 \ --index-url https://download.pytorch.org/whl/cu124 # Install Flash Attention 2 pip install ninja psutil packaging flash_attn==2.7.4.post1 # Install all requirements pip install -r requirements.txt conda install -c conda-forge librosa ffmpeg pip install -r requirements_avatar.txt
# Install the HuggingFace CLI pip install "huggingface_hub[cli]" # Download the base model huggingface-cli download meituan-longcat/LongCat-Video \ --local-dir ./weights/LongCat-Video # Download Avatar 1.5 weights huggingface-cli download meituan-longcat/LongCat-Video-Avatar-1.5 \ --local-dir ./weights/LongCat-Video-Avatar-1.5
# Audio-Text-to-Video (AT2V) torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py \ --context_parallel_size=2 \ --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 \ --stage_1=at2v \ --input_json=assets/avatar/single_example_1.json \ --use_distill --model_type avatar-v1.5 --use_int8 # Audio-Image-to-Video (AI2V) torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py \ --context_parallel_size=2 \ --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 \ --stage_1=ai2v \ --input_json=assets/avatar/single_example_1.json \ --use_distill --model_type avatar-v1.5 --use_int8 # Long video continuation (5 segments) torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py \ --context_parallel_size=2 \ --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 \ --stage_1=at2v --num_segments=5 \ --ref_img_index=10 --mask_frame_range=3 \ --use_distill --model_type avatar-v1.5 --use_int8
# Multi-Person Audio-Image-to-Video torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py \ --context_parallel_size=2 \ --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 \ --input_json=assets/avatar/multi_example_1.json \ --use_distill --model_type avatar-v1.5 --use_int8 # Multi-Person with video continuation torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py \ --context_parallel_size=2 \ --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 \ --input_json=assets/avatar/multi_example_1.json \ --num_segments=5 --ref_img_index=10 --mask_frame_range=3 \ --use_distill --model_type avatar-v1.5 --use_int8

⚙️ Hardware Requirements

Component Minimum (INT8) Recommended
GPU VRAM 16 GB 24–40 GB
System RAM 32 GB 64 GB
Disk Storage 50 GB 100 GB
GPU Model A6000 / 3090 A100 80GB
CUDA Version 12.1+ 12.4
📖 Citation

Cite This Work

@misc{meituanlongcatteam2025longcatvideoavatar15technicalreport, title={LongCat-Video-Avatar 1.5 Technical Report}, author={Meituan LongCat Team}, year={2026}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5}, }