{"record":{"id":"c61ce83c3a321c33","repo":"microsoft/VibeVoice","slug":"unsupported-tokenizer-type-for-language-model-pre-c61ce8","errorCode":null,"errorMessage":"Unsupported tokenizer type for {language_model_pretrained_name}. Supported types: Qwen, Llama, Gemma.","messagePattern":"Unsupported tokenizer type for (.+?)\\. Supported types: Qwen, Llama, Gemma\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vibevoice/processor/vibevoice_processor.py","lineNumber":105,"sourceCode":"                config = {\n                    \"speech_tok_compress_ratio\": 3200,\n                    \"db_normalize\": True,\n                }\n        \n        # Extract main processor parameters\n        speech_tok_compress_ratio = config.get(\"speech_tok_compress_ratio\", 3200)\n        db_normalize = config.get(\"db_normalize\", True)\n        \n        # Load tokenizer - try from model path first, then fall back to Qwen        \n        language_model_pretrained_name = config.get(\"language_model_pretrained_name\", None) or kwargs.pop(\"language_model_pretrained_name\", \"Qwen/Qwen2.5-1.5B\")\n        logger.info(f\"Loading tokenizer from {language_model_pretrained_name}\")\n        if 'qwen' in language_model_pretrained_name.lower():\n            tokenizer = VibeVoiceTextTokenizerFast.from_pretrained(\n                language_model_pretrained_name,\n                **kwargs\n            )\n        else:\n            raise ValueError(f\"Unsupported tokenizer type for {language_model_pretrained_name}. Supported types: Qwen, Llama, Gemma.\")\n        \n        # Load audio processor\n        if \"audio_processor\" in config:\n            # Create audio processor from config\n            audio_config = config[\"audio_processor\"]\n            audio_processor = VibeVoiceTokenizerProcessor(\n                sampling_rate=audio_config.get(\"sampling_rate\", 24000),\n                normalize_audio=audio_config.get(\"normalize_audio\", True),\n                target_dB_FS=audio_config.get(\"target_dB_FS\", -25),\n                eps=audio_config.get(\"eps\", 1e-6),\n            )\n        else:\n            # Create default audio processor\n            audio_processor = VibeVoiceTokenizerProcessor()\n        \n        # Create and return the processor\n        return cls(\n            tokenizer=tokenizer,","sourceCodeStart":87,"sourceCodeEnd":123,"githubUrl":"https://github.com/microsoft/VibeVoice/blob/94da20d98b2fa7688e9cbfaf7692ddb4954f7600/vibevoice/processor/vibevoice_processor.py#L87-L123","documentation":"VibeVoiceProcessor.from_pretrained selects a text tokenizer class by checking for a 'qwen' substring (case-insensitive) in `language_model_pretrained_name`. Despite the error text advertising 'Supported types: Qwen, Llama, Gemma', the code only implements the Qwen branch, so any non-Qwen name raises ValueError. This is a message/implementation mismatch in the library.","triggerScenarios":"Setting language_model_pretrained_name to any string without 'qwen': 'meta-llama/Llama-3-8B', 'google/gemma-7b', or a local path like './lm-checkpoint' (even if it is actually a Qwen save).","commonSituations":"Users believing the error message and trying Llama or Gemma, then hitting the same error; local fine-tuned Qwen checkpoints saved under a name that lost 'qwen'; company-internal mirror hostnames that rename repos.","solutions":["Use 'Qwen/Qwen2.5-1.5B' (default) or another Qwen repo id containing 'qwen'.","For a local Qwen fine-tune, name or symlink the directory so the path contains 'qwen'.","If you truly need Llama/Gemma tokenizers, subclass VibeVoiceProcessor and add the branch; do not rely on the message's claim."],"exampleFix":"# before\nprocessor = VibeVoiceProcessor.from_pretrained(\n    ..., language_model_pretrained_name='meta-llama/Llama-3-8B')\n\n# after\nprocessor = VibeVoiceProcessor.from_pretrained(\n    ..., language_model_pretrained_name='Qwen/Qwen2.5-1.5B')","handlingStrategy":"validation","validationCode":"name = config.get('language_model_pretrained_name') or 'Qwen/Qwen2.5-1.5B'\nassert 'qwen' in name.lower(), (\n    f'Only Qwen tokenizers load; {name!r} lacks \"qwen\" (local dirs must keep it in the path)')","typeGuard":"def is_qwen_tokenizer_name(name: str) -> bool:\n    return isinstance(name, str) and 'qwen' in name.lower()","tryCatchPattern":"try:\n    processor = VibeVoiceProcessor.from_pretrained(model_path)\nexcept ValueError as e:\n    if 'Unsupported tokenizer type' in str(e):\n        raise ValueError('Set language_model_pretrained_name to a Qwen checkpoint; '\n                         'Llama/Gemma are NOT actually wired up despite the message') from e\n    raise","preventionTips":["Do not trust the error message's claim that Llama/Gemma work — only Qwen is implemented.","Keep 'qwen' in the path of local fine-tunes or symlink accordingly.","Validate the tokenizer name before from_pretrained to give a clearer upstream error."],"tags":["tokenizer","config","from-pretrained","misleading-error"],"backgroundTag":null,"analyzedSha":"94da20d98b2fa7688e9cbfaf7692ddb4954f7600","analyzedAt":"2026-08-15T04:12:07.418Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}