{"record":{"id":"5b25deaa65fe8995","repo":"microsoft/VibeVoice","slug":"speech-type-speech-type-not-implemented","errorCode":null,"errorMessage":"Speech type {speech_type} not implemented","messagePattern":"Speech type (.+?) not implemented","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"vibevoice/modular/modeling_vibevoice.py","lineNumber":306,"sourceCode":"            with torch.no_grad():\n                if speech_type == \"audio\":\n                    with torch.no_grad():\n                        frames = self.model.acoustic_tokenizer.encode(speech_tensors.unsqueeze(1))[0][0]\n                    audio_tokens = frames.sample(self.model.acoustic_tokenizer.std_dist_type)[0]\n\n                elif speech_type == \"vae\":\n                    # Use config to get vae_dim instead of non-existent self.args\n                    vae_dim = self.config.acoustic_tokenizer_config.vae_dim\n                    speech_mode = speech_tensors.reshape(speech_tensors.size(0), -1, vae_dim)\n\n                    # gaussian sample from the speech_mode\n                    batch_size = speech_mode.size(0)\n                    value = self.model.acoustic_tokenizer.fix_std / 0.8\n                    std = torch.randn(batch_size, dtype=speech_mode.dtype, device=speech_mode.device) * value\n                    std = std.view(-1, *[1] * (speech_mode.dim() - 1))\n                    audio_tokens = speech_mode + std * torch.randn(speech_mode.shape).to(speech_mode)\n                else:\n                    raise NotImplementedError(f\"Speech type {speech_type} not implemented\")\n                \n                if torch.isnan(self.model.speech_scaling_factor) or torch.isnan(self.model.speech_bias_factor):\n                    scaling_factor = 1. / audio_tokens[speech_masks].flatten().std()\n                    bias_factor = -audio_tokens[speech_masks].flatten().mean()\n                    \n                    # Only use distributed operations if the process group is initialized\n                    if dist.is_available() and dist.is_initialized():\n                        dist.all_reduce(scaling_factor, op=dist.ReduceOp.SUM)\n                        dist.all_reduce(bias_factor, op=dist.ReduceOp.SUM)\n                        world_size = dist.get_world_size()\n                        self.model.speech_scaling_factor.copy_(scaling_factor / world_size)  \n                        self.model.speech_bias_factor.copy_(bias_factor / world_size)\n                        print(f\"Speech scaling factor (distributed): {self.model.speech_scaling_factor}, bias factor: {self.model.speech_bias_factor}\", flush=True)\n                    else:\n                        # Single process case\n                        self.model.speech_scaling_factor.copy_(scaling_factor)  \n                        self.model.speech_bias_factor.copy_(bias_factor)\n                        print(f\"Speech scaling factor (single process): {self.model.speech_scaling_factor}, bias factor: {self.model.speech_bias_factor}\", flush=True)","sourceCodeStart":288,"sourceCodeEnd":324,"githubUrl":"https://github.com/microsoft/VibeVoice/blob/94da20d98b2fa7688e9cbfaf7692ddb4954f7600/vibevoice/modular/modeling_vibevoice.py#L288-L324","documentation":"In the data collator path of modeling_vibevoice.py, speech input can be supplied as pre-extracted tokens ('tokens') or continuous VAE latents ('vae'); any other speech_type string raises NotImplementedError. It marks the boundary of supported training-data formats for speech representation.","triggerScenarios":"A dataset/dataloader passes speech_type other than 'tokens' or 'vae' (e.g. 'raw', 'mel', typo like 'VAE' with different casing per the branch checks) while speech_tensors is present.","commonSituations":"Custom training data with a new modality field; dataset config renamed the speech_type field value; casing/whitespace mismatch in the string coming from JSON metadata.","solutions":["Set the dataset's speech_type to 'tokens' or 'vae' — the only implemented formats.","Check exact casing/whitespace of the value read from metadata and normalize it before batching.","If you genuinely need a new representation, implement a branch in modeling_vibevoice.py before the raise and contribute it.","Log speech_type per batch during dataset debugging to catch the offending sample."],"exampleFix":"# before\n{\"speech_type\": \"raw\", \"speech\": wav}  # -> NotImplementedError\n\n# after\nfrom vibevoice.processor import ...\nspeech = processor(wav)  # pre-tokenize\n{\"speech_type\": \"tokens\", \"speech\": speech}","handlingStrategy":"validation","validationCode":"assert speech_type in {\"tokens\", \"vae\"}, f\"Unsupported speech_type {speech_type!r}\"","typeGuard":"def is_supported_speech_type(s: object) -> bool:\n    return s in (\"tokens\", \"vae\")","tryCatchPattern":"try:\n    batch = collator(features)\nexcept NotImplementedError as e:\n    raise ValueError(f\"Bad sample in batch: {e}\") from e","preventionTips":["Normalize speech_type strings (strip, lower) when reading dataset metadata","Pre-tokenize audio upstream so datasets always use 'tokens'","Log offending sample indices when collation fails"],"tags":["training","collator","speech-type","notimplementederror"],"backgroundTag":null,"analyzedSha":"94da20d98b2fa7688e9cbfaf7692ddb4954f7600","analyzedAt":"2026-08-15T04:12:07.418Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}