microsoft/VibeVoice · error · NotImplementedError

Speech type {speech_type} not implemented

Error message

Speech type {speech_type} not implemented

What it means

In the data collator path of modeling_vibevoice.py, speech input can be supplied as pre-extracted tokens ('tokens') or continuous VAE latents ('vae'); any other speech_type string raises NotImplementedError. It marks the boundary of supported training-data formats for speech representation.

Source

Thrown at vibevoice/modular/modeling_vibevoice.py:306

            with torch.no_grad():
                if speech_type == "audio":
                    with torch.no_grad():
                        frames = self.model.acoustic_tokenizer.encode(speech_tensors.unsqueeze(1))[0][0]
                    audio_tokens = frames.sample(self.model.acoustic_tokenizer.std_dist_type)[0]

                elif speech_type == "vae":
                    # Use config to get vae_dim instead of non-existent self.args
                    vae_dim = self.config.acoustic_tokenizer_config.vae_dim
                    speech_mode = speech_tensors.reshape(speech_tensors.size(0), -1, vae_dim)

                    # gaussian sample from the speech_mode
                    batch_size = speech_mode.size(0)
                    value = self.model.acoustic_tokenizer.fix_std / 0.8
                    std = torch.randn(batch_size, dtype=speech_mode.dtype, device=speech_mode.device) * value
                    std = std.view(-1, *[1] * (speech_mode.dim() - 1))
                    audio_tokens = speech_mode + std * torch.randn(speech_mode.shape).to(speech_mode)
                else:
                    raise NotImplementedError(f"Speech type {speech_type} not implemented")
                
                if torch.isnan(self.model.speech_scaling_factor) or torch.isnan(self.model.speech_bias_factor):
                    scaling_factor = 1. / audio_tokens[speech_masks].flatten().std()
                    bias_factor = -audio_tokens[speech_masks].flatten().mean()
                    
                    # Only use distributed operations if the process group is initialized
                    if dist.is_available() and dist.is_initialized():
                        dist.all_reduce(scaling_factor, op=dist.ReduceOp.SUM)
                        dist.all_reduce(bias_factor, op=dist.ReduceOp.SUM)
                        world_size = dist.get_world_size()
                        self.model.speech_scaling_factor.copy_(scaling_factor / world_size)  
                        self.model.speech_bias_factor.copy_(bias_factor / world_size)
                        print(f"Speech scaling factor (distributed): {self.model.speech_scaling_factor}, bias factor: {self.model.speech_bias_factor}", flush=True)
                    else:
                        # Single process case
                        self.model.speech_scaling_factor.copy_(scaling_factor)  
                        self.model.speech_bias_factor.copy_(bias_factor)
                        print(f"Speech scaling factor (single process): {self.model.speech_scaling_factor}, bias factor: {self.model.speech_bias_factor}", flush=True)

View on GitHub (pinned to 94da20d98b)

Solutions

1. Set the dataset's speech_type to 'tokens' or 'vae' — the only implemented formats.
2. Check exact casing/whitespace of the value read from metadata and normalize it before batching.
3. If you genuinely need a new representation, implement a branch in modeling_vibevoice.py before the raise and contribute it.
4. Log speech_type per batch during dataset debugging to catch the offending sample.

Example fix

# before
{"speech_type": "raw", "speech": wav}  # -> NotImplementedError

# after
from vibevoice.processor import ...
speech = processor(wav)  # pre-tokenize
{"speech_type": "tokens", "speech": speech}

assert speech_type in {"tokens", "vae"}, f"Unsupported speech_type {speech_type!r}"

def is_supported_speech_type(s: object) -> bool:
    return s in ("tokens", "vae")

try:
    batch = collator(features)
except NotImplementedError as e:
    raise ValueError(f"Bad sample in batch: {e}") from e

When it happens

Trigger: A dataset/dataloader passes speech_type other than 'tokens' or 'vae' (e.g. 'raw', 'mel', typo like 'VAE' with different casing per the branch checks) while speech_tensors is present.

Common situations: Custom training data with a new modality field; dataset config renamed the speech_type field value; casing/whitespace mismatch in the string coming from JSON metadata.

Related errors

• Prediction type {prediction_type} not implemented
• Loss computation is not implemented in this version.

AI-assisted analysis of microsoft/VibeVoice@94da20d98b (2026-08-15). Data as JSON: /api/errors/5b25deaa65fe8995. Report an issue: GitHub.