mudler/LocalAI · error · ValueError

audio encoder returned non-finite values

Error message

audio encoder returned non-finite values

What it means

ValueError after pipeline.get_audio_embedding(...): the avatar audio encoder produced a tensor containing NaN or Inf (torch.isfinite(...).all() fails). Non-finite embeddings propagate into diffusion conditioning and poison every generated frame, so the backend refuses to continue. Root causes are almost always upstream: corrupted/clipped audio, silent-after-padding edge cases, a broken float16 encoder, or GPU numerical issues.

Source

Thrown at backend/python/longcat-video/backend.py:677

        avatar_fps = 25
        generated_duration = (
            segment_frames + (segments - 1) * (segment_frames - conditioning_frames)
        ) / avatar_fps
        pad_samples = max(
            0, math.ceil((generated_duration - audio_duration) * sample_rate)
        )
        if pad_samples:
            speech = self.np.pad(speech, (0, pad_samples))

        full_audio_embedding = self.pipeline.get_audio_embedding(
            speech,
            fps=avatar_fps,
            device=self.device_index,
            sample_rate=sample_rate,
            model_type="avatar-v1.5",
        )
        if not self.torch.isfinite(full_audio_embedding).all():
            raise ValueError("audio encoder returned non-finite values")

        indices = self.torch.arange(5) - 2

        def audio_window(start_index):
            centers = self.torch.arange(
                start_index,
                start_index + segment_frames,
            ).unsqueeze(1) + indices.unsqueeze(0)
            centers = self.torch.clamp(
                centers,
                min=0,
                max=full_audio_embedding.shape[0] - 1,
            )
            return full_audio_embedding[centers][None, ...].to(self.device_index)

        audio_start = 0
        common = {
            "prompt": request.prompt,

View on GitHub (pinned to 44413a9d06)

Solutions

1. Normalize/clean the audio first: peak-normalize, remove long silences, ensure 16 kHz mono wav
2. Retry with a different audio file to determine whether the input or the model is at fault
3. If every input fails: re-download the avatar model weights and check GPU health (dmesg for Xid/ECC errors, run with float32)

Example fix

# before
speech, sr = librosa.load(path, sr=16000, mono=True)

# after
speech, sr = librosa.load(path, sr=16000, mono=True)
import numpy as np
speech = speech / (np.max(np.abs(speech)) + 1e-9)  # peak-normalize before sending

import numpy as np, librosa

def safe_load_speech(path: str, sr: int = 16000):
    speech, rate = librosa.load(path, sr=sr, mono=True)
    if speech.size == 0:
        raise ValueError("empty audio")
    peak = np.max(np.abs(speech))
    if not np.isfinite(speech).all() or peak > 1.0:
        speech = np.nan_to_num(speech)
        speech = speech / (peak + 1e-9)
    return speech, rate

try:
    stub.GenerateVideo(req)
except grpc.RpcError as e:
    if "non-finite" in (e.details() or ""):
        # input-side mitigation first: normalize + strip silence, then one retry
        req.audio = normalize_and_restage(req.audio)
        stub.GenerateVideo(req)
    else:
        raise

When it happens

Trigger: Extreme-amplitude or DC-offset audio fed to the encoder; audio with only silence after trimming; GPU in a bad state (ECC errors, unstable clocks); mismatched model_type/sample_rate inputs to get_audio_embedding.

Common situations: Raw unnormalized recordings straight from a mic; fp16 overflow in the audio encoder on certain GPUs; corrupted download of the avatar audio encoder weights.

Related errors

• audio is required for LongCat-Video-Avatar-1.5
• audio input is not a readable staged file
• audio contains no samples
• request needs {segments} avatar segments, but max_segments i
• resolution must be 480p or 720p

AI-assisted analysis of mudler/LocalAI@44413a9d06 (2026-08-15). Data as JSON: /api/errors/ede290cbf642b149. Report an issue: GitHub.