{"record":{"id":"ede290cbf642b149","repo":"mudler/LocalAI","slug":"audio-encoder-returned-non-finite-values","errorCode":null,"errorMessage":"audio encoder returned non-finite values","messagePattern":"audio encoder returned non-finite values","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"backend/python/longcat-video/backend.py","lineNumber":677,"sourceCode":"        avatar_fps = 25\n        generated_duration = (\n            segment_frames + (segments - 1) * (segment_frames - conditioning_frames)\n        ) / avatar_fps\n        pad_samples = max(\n            0, math.ceil((generated_duration - audio_duration) * sample_rate)\n        )\n        if pad_samples:\n            speech = self.np.pad(speech, (0, pad_samples))\n\n        full_audio_embedding = self.pipeline.get_audio_embedding(\n            speech,\n            fps=avatar_fps,\n            device=self.device_index,\n            sample_rate=sample_rate,\n            model_type=\"avatar-v1.5\",\n        )\n        if not self.torch.isfinite(full_audio_embedding).all():\n            raise ValueError(\"audio encoder returned non-finite values\")\n\n        indices = self.torch.arange(5) - 2\n\n        def audio_window(start_index):\n            centers = self.torch.arange(\n                start_index,\n                start_index + segment_frames,\n            ).unsqueeze(1) + indices.unsqueeze(0)\n            centers = self.torch.clamp(\n                centers,\n                min=0,\n                max=full_audio_embedding.shape[0] - 1,\n            )\n            return full_audio_embedding[centers][None, ...].to(self.device_index)\n\n        audio_start = 0\n        common = {\n            \"prompt\": request.prompt,","sourceCodeStart":659,"sourceCodeEnd":695,"githubUrl":"https://github.com/mudler/LocalAI/blob/44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26/backend/python/longcat-video/backend.py#L659-L695","documentation":"ValueError after pipeline.get_audio_embedding(...): the avatar audio encoder produced a tensor containing NaN or Inf (torch.isfinite(...).all() fails). Non-finite embeddings propagate into diffusion conditioning and poison every generated frame, so the backend refuses to continue. Root causes are almost always upstream: corrupted/clipped audio, silent-after-padding edge cases, a broken float16 encoder, or GPU numerical issues.","triggerScenarios":"Extreme-amplitude or DC-offset audio fed to the encoder; audio with only silence after trimming; GPU in a bad state (ECC errors, unstable clocks); mismatched model_type/sample_rate inputs to get_audio_embedding.","commonSituations":"Raw unnormalized recordings straight from a mic; fp16 overflow in the audio encoder on certain GPUs; corrupted download of the avatar audio encoder weights.","solutions":["Normalize/clean the audio first: peak-normalize, remove long silences, ensure 16 kHz mono wav","Retry with a different audio file to determine whether the input or the model is at fault","If every input fails: re-download the avatar model weights and check GPU health (dmesg for Xid/ECC errors, run with float32)"],"exampleFix":"# before\nspeech, sr = librosa.load(path, sr=16000, mono=True)\n\n# after\nspeech, sr = librosa.load(path, sr=16000, mono=True)\nimport numpy as np\nspeech = speech / (np.max(np.abs(speech)) + 1e-9)  # peak-normalize before sending","handlingStrategy":"try-catch","validationCode":"import numpy as np, librosa\n\ndef safe_load_speech(path: str, sr: int = 16000):\n    speech, rate = librosa.load(path, sr=sr, mono=True)\n    if speech.size == 0:\n        raise ValueError(\"empty audio\")\n    peak = np.max(np.abs(speech))\n    if not np.isfinite(speech).all() or peak > 1.0:\n        speech = np.nan_to_num(speech)\n        speech = speech / (peak + 1e-9)\n    return speech, rate","typeGuard":null,"tryCatchPattern":"try:\n    stub.GenerateVideo(req)\nexcept grpc.RpcError as e:\n    if \"non-finite\" in (e.details() or \"\"):\n        # input-side mitigation first: normalize + strip silence, then one retry\n        req.audio = normalize_and_restage(req.audio)\n        stub.GenerateVideo(req)\n    else:\n        raise","preventionTips":["Peak-normalize and validate audio with numpy before staging","If it reproduces across all inputs, suspect model weights or GPU health rather than the audio"],"tags":["python","longcat-video","audio","numerical","gpu"],"backgroundTag":null,"analyzedSha":"44413a9d06bf5bc52ce088ba8ca74e5a2e8bee26","analyzedAt":"2026-08-15T10:13:50.291Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}