microsoft/VibeVoice · error · ValueError

Unsupported audio data type: {type(data)}

Error message

Unsupported audio data type: {type(data)}

What it means

Raised by the vLLM multimodal input parser for VibeVoice when the audio payload is not one of the supported types. The branch chain in vllm_plugin/inputs.py:82 only handles a path/str (via load_audio), bytes (via ffmpeg stdin decode + AudioNormalizer), and numpy.ndarray (used as-is). Any other object type falls through to this ValueError before any tensor conversion happens.

Source

Thrown at vllm_plugin/inputs.py:82

        data = data[0]

    audio_waveform = None
    
    if isinstance(data, str):
        # Load from file path
        audio_waveform = load_audio(data)
        
    elif isinstance(data, bytes):
        # Decode bytes directly via ffmpeg stdin pipe to avoid temp-file IO
        audio_waveform, _sr = load_audio_bytes_use_ffmpeg(data, resample=True, target_sr=24000)
        normalizer = AudioNormalizer()
        audio_waveform = normalizer(audio_waveform)
                
    elif isinstance(data, np.ndarray):
        # Already loaded numpy array
        audio_waveform = data
    else:
        raise ValueError(f"Unsupported audio data type: {type(data)}")

    # Validate audio duration before tensor conversion to catch OOM early
    duration_sec = len(audio_waveform) / 24000
    if duration_sec > _MAX_AUDIO_DURATION:
        raise ValueError(
            f"Audio duration ({duration_sec:.1f}s) exceeds the configured "
            f"limit ({_MAX_AUDIO_DURATION:.0f}s). Set the "
            f"VIBEVOICE_MAX_AUDIO_DURATION environment variable to adjust "
            f"this limit, or use shorter audio."
        )

    # Convert to tensor
    audio_tensor = torch.from_numpy(audio_waveform).float()
    audio_length = audio_tensor.shape[0]
    
    return MultiModalInputs({
        "audio": audio_tensor,
        "audio_length": audio_length

View on GitHub (pinned to 94da20d98b)

Solutions

1. Convert the payload before submitting: waveforms to numpy float32 mono at 24 kHz, or raw bytes of any audio file, or a file path string.
2. If you have a torch.Tensor, call `tensor.numpy()` (after `.cpu().detach()`); if you have (waveform, sr) from a loader, pass only the waveform and resample to 24000 first.
3. If base64 audio arrives from an API, decode with base64.b64decode(...) and send the resulting bytes.
4. If a batch, submit each array as a separate audio item rather than a list under one item.

Example fix

// before
inputs = {"audio": (waveform, 24000)}  # tuple from torchaudio.load

# after
waveform = torchaudio.functional.resample(waveform, orig_sr, 24000)
inputs = {"audio": waveform.numpy()}  # bare np.ndarray, 24 kHz mono

import numpy as np
def coerce_audio(data):
    if isinstance(data, str):
        return data                      # path: handled by load_audio
    if isinstance(data, (bytes, bytearray)):
        return bytes(data)               # ffmpeg stdin path
    if isinstance(data, np.ndarray) and data.ndim == 1:
        return data
    if str(type(data)) == "<class 'torch.Tensor'>":
        return data.detach().cpu().numpy()  # common caller mistake
    raise TypeError(f"Convert audio to str path / bytes / 1-D np.ndarray, got {type(data)}")

def is_supported_audio(data) -> bool:
    return isinstance(data, (str, bytes, bytearray)) or (
        isinstance(data, np.ndarray) and data.ndim == 1
    )

try:
    mm = plugin_input_parser(data)
except ValueError as e:
    if "Unsupported audio data type" in str(e):
        data = coerce_audio(data)  # then retry once
    else:
        raise

When it happens

Trigger: Passing audio as a torch.Tensor, a Python list of floats, a dict (e.g. {'audio': ...}), a URL, a BytesIO/file object, or None in the 'audio' field of a multimodal input dict. Also triggered by passing a numpy array subclass that fails isinstance(data, np.ndarray) is unlikely, but wrapping audio in an extra layer (list of arrays for a batch) hits it.

Common situations: Callers coming from other vLLM multimodal models (e.g. Qwen-Audio) that accept torch tensors or pre-encoded features; sending JSON payloads where audio arrives as base64 str without decoding; accidentally sending a tuple of (waveform, sample_rate) returned by librosa/torchaudio load instead of the waveform alone.

Related errors

• Audio input is required for ASR processing
• Unexpected audio shape: {audio.shape}
• Audio should be 1D or 2D, got shape: {audio.shape}
• Audio input is required
• Audio duration ({duration_sec:.1f}s) exceeds the configured

AI-assisted analysis of microsoft/VibeVoice@94da20d98b (2026-08-15). Data as JSON: /api/errors/7e6ca13c06c0ef14. Report an issue: GitHub.