{"record":{"id":"45a68e5be42e96fd","repo":"huggingface/transformers","slug":"incorrect-format-used-for-audio-should-be-a-num","errorCode":null,"errorMessage":"Incorrect format used for `audio`. Should be a numpy array or a `str`: an `http(s)://` URL, a local file path, or a base64-encoded string (optionally wrapped as a `data:...` URI).","messagePattern":"Incorrect format used for `audio`\\. Should be a numpy array or a `str`: an `http\\(s\\)://` URL, a local file path, or a base64-encoded string \\(optionally wrapped as a `data:\\.\\.\\.` URI\\)\\.","errorType":"exception","errorClass":"TypeError","httpStatus":null,"severity":"error","filePath":"src/transformers/audio_utils.py","lineNumber":224,"sourceCode":"            `data:<media-type>;base64,` URI).\n        sampling_rate (`int`, *optional*, defaults to 16000):\n            The sampling rate to be used when loading the audio. It should be same as the\n            sampling rate the model you will be using further was trained with.\n        timeout (`float`, *optional*):\n            The timeout value in seconds for the URL request.\n        backend (`str`, *optional*, defaults to `\"auto\"`):\n            Decoding backend: `\"auto\"` uses torchcodec when available (>=0.3.0) and falls back to\n            librosa; `\"torchcodec\"`, `\"librosa\"` or `\"torchaudio\"` force that backend (and error if it\n            is missing). `\"torchaudio\"` decodes with `torchaudio.load` and resamples with\n            `torchaudio.functional.resample` (matches serving stacks such as sglang bit-for-bit).\n\n    Returns:\n        `np.ndarray`: A numpy array representing the audio.\n    \"\"\"\n    if isinstance(audio, np.ndarray):\n        return audio\n    if not isinstance(audio, str):\n        raise TypeError(\n            \"Incorrect format used for `audio`. Should be a numpy array or a `str`: an `http(s)://` URL, \"\n            \"a local file path, or a base64-encoded string (optionally wrapped as a `data:...` URI).\"\n        )\n\n    # torchcodec handles audio/video; librosa only plain audio. `backend` lets callers pin one.\n    if backend == \"auto\":\n        resolved_backend = (\n            \"torchcodec\" if is_torchcodec_available() and version.parse(\"0.3.0\") <= TORCHCODEC_VERSION else \"librosa\"\n        )\n    elif backend in (\"torchcodec\", \"librosa\", \"torchaudio\"):\n        resolved_backend = backend\n    else:\n        raise ValueError(f\"Unknown backend {backend!r}; expected 'auto', 'torchcodec', 'librosa', or 'torchaudio'.\")\n    # soundfile-based backends (librosa / torchaudio) cannot decode the video-ish formats below.\n    use_torchcodec = resolved_backend == \"torchcodec\"\n\n    # 1. Identify the format from the source string (extension / `data:` media type), without fetching.\n    filetype = _format_from_source(audio)","sourceCodeStart":206,"sourceCodeEnd":242,"githubUrl":"https://github.com/huggingface/transformers/blob/a597f974857b3d92939971296bc0deb93d33d780/src/transformers/audio_utils.py#L206-L242","documentation":"Identical dtype gate to the BF16 experts path, applied in `deepgemm_fp8_fp4_experts_forward`: hidden states entering the FP8/FP4 grouped experts must be bfloat16, because the per-token cast and the grouped kernels' dequant output are bf16. fp16/fp32 activations are rejected before kernel load.","triggerScenarios":"FP8/FP4 DeepGEMM experts forward with `hidden_states.dtype != torch.bfloat16` — typically a model loaded in float16 or float32, or a preceding layer that upcasts to float32 (e.g. RMSNorm in fp32 without a downcast).","commonSituations":"`torch_dtype=torch.float16` checkpoints on new stacks; custom norm code that leaves `.float()` activations; mixed autocast configurations where the MoE block runs outside the autocast region.","solutions":["Load/run the model in bfloat16 (`torch_dtype=torch.bfloat16`)","Cast at the boundary: `hidden_states = hidden_states.to(torch.bfloat16)` before the experts","Verify autocast dtype is bf16 when the MoE forward runs under autocast"],"exampleFix":"# before\nout = model(input_ids)  # model in fp16 -> ValueError\n\n# after\nmodel = AutoModelForCausalLM.from_pretrained(ckpt, torch_dtype=torch.bfloat16)\nout = model(input_ids)","handlingStrategy":"type-guard","validationCode":"if hidden_states.dtype != torch.bfloat16:\n    hidden_states = hidden_states.to(torch.bfloat16)","typeGuard":"def is_bf16(t: torch.Tensor) -> bool:\n    return t.dtype == torch.bfloat16","tryCatchPattern":null,"preventionTips":["Keep the whole MoE stack in bf16 (load with torch_dtype=torch.bfloat16)","Watch for norms/upcasts that hand fp32 activations to the experts"],"tags":["deepgemm","bfloat16","moe","dtype"],"backgroundTag":null,"analyzedSha":"a597f974857b3d92939971296bc0deb93d33d780","analyzedAt":"2026-08-14T18:24:08.354Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}