{"record":{"id":"04a40afa8247b0a2","repo":"microsoft/VibeVoice","slug":"unsupported-tokenizer-type-for-language-model-pre-04a40a","errorCode":null,"errorMessage":"Unsupported tokenizer type for {language_model_pretrained_name}. Supported types: Qwen, Llama, Gemma.","messagePattern":"Unsupported tokenizer type for (.+?)\\. Supported types: Qwen, Llama, Gemma\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vibevoice/processor/vibevoice_streaming_processor.py","lineNumber":101,"sourceCode":"                config = {\n                    \"speech_tok_compress_ratio\": 3200,\n                    \"db_normalize\": True,\n                }\n        \n        # Extract main processor parameters\n        speech_tok_compress_ratio = config.get(\"speech_tok_compress_ratio\", 3200)\n        db_normalize = config.get(\"db_normalize\", True)\n        \n        # Load tokenizer - try from model path first, then fall back to Qwen        \n        language_model_pretrained_name = config.get(\"language_model_pretrained_name\", None) or kwargs.pop(\"language_model_pretrained_name\", \"Qwen/Qwen2.5-1.5B\")\n        logger.info(f\"Loading tokenizer from {language_model_pretrained_name}\")\n        if 'qwen' in language_model_pretrained_name.lower():\n            tokenizer = VibeVoiceTextTokenizerFast.from_pretrained(\n                language_model_pretrained_name,\n                **kwargs\n            )\n        else:\n            raise ValueError(f\"Unsupported tokenizer type for {language_model_pretrained_name}. Supported types: Qwen, Llama, Gemma.\")\n        \n        # Load audio processor\n        if \"audio_processor\" in config:\n            # Create audio processor from config\n            audio_config = config[\"audio_processor\"]\n            audio_processor = VibeVoiceTokenizerProcessor(\n                sampling_rate=audio_config.get(\"sampling_rate\", 24000),\n                normalize_audio=audio_config.get(\"normalize_audio\", True),\n                target_dB_FS=audio_config.get(\"target_dB_FS\", -25),\n                eps=audio_config.get(\"eps\", 1e-6),\n            )\n        else:\n            # Create default audio processor\n            audio_processor = VibeVoiceTokenizerProcessor()\n        \n        # Create and return the processor\n        return cls(\n            tokenizer=tokenizer,","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/microsoft/VibeVoice/blob/94da20d98b2fa7688e9cbfaf7692ddb4954f7600/vibevoice/processor/vibevoice_streaming_processor.py#L83-L119","documentation":"VibeVoiceStreamingProcessor.from_pretrained uses the same tokenizer-selection logic as the non-streaming processor: a case-insensitive 'qwen' substring check on language_model_pretrained_name, with a single implemented branch. The advertised 'Supported types: Qwen, Llama, Gemma' does not match the code — only Qwen actually loads.","triggerScenarios":"Constructing the streaming processor with language_model_pretrained_name lacking 'qwen': Llama/Gemma repo ids, or a local directory like '/srv/models/lm' that contains a Qwen tokenizer but no 'qwen' in the path.","commonSituations":"Switching an inference server from VibeVoiceProcessor to VibeVoiceStreamingProcessor while reusing a renamed local checkpoint path; on-prem mirrors that strip the org name.","solutions":["Set language_model_pretrained_name to a Qwen repo id (default 'Qwen/Qwen2.5-1.5B') or keep 'qwen' in the local path name.","Symlink or rename local fine-tune directories so the substring check passes.","Subclass and extend the branch if a different tokenizer family is genuinely required."],"exampleFix":"# before\nproc = VibeVoiceStreamingProcessor.from_pretrained('/srv/models/vibevoice-1.5B',\n    language_model_pretrained_name='/srv/models/lm')\n\n# after\nproc = VibeVoiceStreamingProcessor.from_pretrained('/srv/models/vibevoice-1.5B',\n    language_model_pretrained_name='Qwen/Qwen2.5-1.5B')","handlingStrategy":"validation","validationCode":"name = config.get('language_model_pretrained_name') or 'Qwen/Qwen2.5-1.5B'\nif 'qwen' not in name.lower():\n    name = 'Qwen/Qwen2.5-1.5B'  # or fail loudly: raise ValueError(...)\nprocessor = VibeVoiceStreamingProcessor.from_pretrained(\n    ..., language_model_pretrained_name=name)","typeGuard":"def is_qwen_tokenizer_name(name: str) -> bool:\n    return isinstance(name, str) and 'qwen' in name.lower()","tryCatchPattern":"try:\n    proc = VibeVoiceStreamingProcessor.from_pretrained(model_path)\nexcept ValueError as e:\n    if 'Unsupported tokenizer type' in str(e):\n        proc = VibeVoiceStreamingProcessor.from_pretrained(\n            model_path, language_model_pretrained_name='Qwen/Qwen2.5-1.5B')\n    else:\n        raise","preventionTips":["Treat 'qwen' in the checkpoint path as a hard requirement when configuring streaming.","Store tokenizer repo ids as constants rather than derived paths.","Recheck the substring condition after renaming or migrating model directories."],"tags":["tokenizer","streaming","config","misleading-error"],"backgroundTag":null,"analyzedSha":"94da20d98b2fa7688e9cbfaf7692ddb4954f7600","analyzedAt":"2026-08-15T04:12:07.418Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}