Audio-to-Video

This guide walks you through the Audio-to-Video workflow: a sample ComfyUI workflow from LTX that generates video timed to an input audio track. You supply an audio clip, and the workflow generates a video whose motion and pacing follow the sound. An optional first-frame image sets the opening shot.

This guide assumes you’re comfortable prompting, generating, and iterating in ComfyUI. If you’re new to LTX in ComfyUI, start with the Text-to-Video guide.

What You’ll Need

Download the Audio-to-Video workflow and drag it into ComfyUI. Open the Workflow Overview panel; if any custom nodes or model files are missing, it will let you install or download them.

Model files:

Download the LTX-2.5 weights from the LTX-2.5 HuggingFace repository (click Agree and Access on first download).

Step-by-Step

1. Load your audio

Load your source clip in the LoadAudio node. In the TrimAudioDuration node, set the start time and duration to select the segment to condition on. The generated video’s length follows this audio, so trim to exactly the section you want.

2. Set a first frame (optional)

To open on a specific image, load it in the LoadImage node and enable the image input. With no image, the workflow generates the opening shot from your prompt alone.

3. Add your prompt

The audio drives timing and motion; this prompt supplies the visual content. Enable enhance positive prompt to expand a short prompt with the Gemma 4 enhancer (the negative prompt is never enhanced). See the Prompting Guide for additional info.

4. Generate

Click Run. The clip is encoded and held fixed while the model generates video against it (see How It Works). Stage 1 samples at the base resolution; Stage 2 upscales 2× and refines. Your original audio track is saved with the generated video, unchanged.

5. Review and iterate

The output is an MP4 carrying your original audio track. To iterate: adjust the prompt, change the trimmed audio segment, swap the first-frame image, or try a different resolution.

How It Works

1. Encode — the trimmed clip (start + duration) is encoded to latent tokens by the audio VAE (LTXVAudioVAEEncode).
2. Freeze — those audio tokens are concatenated with the video latent (LTXVConcatAVLatent) and held frozen through both stages, so the model generates video to match fixed audio rather than generating new audio.
3. Generate — Stage 1 samples the video at the base resolution; Stage 2 upscales 2× and refines. LTXVSeparateAVLatent then splits the result and only the video branch is decoded.
4. Output — the video branch is decoded (VAEDecodeTiled); the SaveVideo node then writes that video out with your original (trimmed) audio track attached. There’s no audio decode step, so the soundtrack is preserved as supplied.

Customization

Trim

The video length is derived from the trimmed audio, so use TrimAudioDuration’s start and duration to set both the segment and the clip length. The frame count still resolves to 1 + a multiple of 8, so the final duration may land slightly off the trimmed length.

Tiled VAE decode

Decoding is tiled to reduce peak VRAM. Adjust tile count and overlap only if you hit memory issues during decode (fewer, larger tiles run faster but need more memory).

CFG

Both stages use CFG 1. The distilled model bakes guidance into distillation, so raising CFG doesn’t improve output the way it might with a standard diffusion model. If you experiment, stay near 1.

Python

Audio-to-video is also available through the PyTorch API as A2VidPipelineTwoStage, for programmatic use and custom pipelines. See the PyTorch API documentation.