{"record":{"id":"104059ac09b94ced","repo":"microsoft/VibeVoice","slug":"unsupported-tokenizer-type-for-language-model-pre","errorCode":null,"errorMessage":"Unsupported tokenizer type for {language_model_pretrained_name}","messagePattern":"Unsupported tokenizer type for (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"vibevoice/processor/vibevoice_asr_processor.py","lineNumber":147,"sourceCode":"                logger.warning(f\"Could not load preprocessor_config.json: {e}\")\n                logger.warning(\"Using default configuration\")\n        \n        # Extract parameters\n        speech_tok_compress_ratio = config.get(\"speech_tok_compress_ratio\", 3200)\n        target_sample_rate = config.get(\"target_sample_rate\", 24000)\n        normalize_audio = config.get(\"normalize_audio\", True)\n        \n        # Load tokenizer\n        language_model_pretrained_name = config.get(\"language_model_pretrained_name\", None) or kwargs.pop(\"language_model_pretrained_name\", \"Qwen/Qwen2.5-1.5B\")\n        logger.info(f\"Loading tokenizer from {language_model_pretrained_name}\")\n        \n        if 'qwen' in language_model_pretrained_name.lower():\n            tokenizer = VibeVoiceASRTextTokenizerFast.from_pretrained(\n                language_model_pretrained_name,\n                **kwargs\n            )\n        else:\n            raise ValueError(f\"Unsupported tokenizer type for {language_model_pretrained_name}\")\n        \n        # Load audio processor\n        audio_processor = VibeVoiceTokenizerProcessor(\n            sampling_rate=target_sample_rate,\n            normalize_audio=normalize_audio,\n            target_dB_FS=config.get(\"target_dB_FS\", -25),\n            eps=config.get(\"eps\", 1e-6),\n        )\n        \n        return cls(\n            tokenizer=tokenizer,\n            audio_processor=audio_processor,\n            speech_tok_compress_ratio=speech_tok_compress_ratio,\n            target_sample_rate=target_sample_rate,\n            normalize_audio=normalize_audio,\n        )\n    \n    def save_pretrained(self, save_directory: Union[str, os.PathLike], **kwargs):","sourceCodeStart":129,"sourceCodeEnd":165,"githubUrl":"https://github.com/microsoft/VibeVoice/blob/94da20d98b2fa7688e9cbfaf7692ddb4954f7600/vibevoice/processor/vibevoice_asr_processor.py#L129-L165","documentation":"VibeVoiceASRProcessor.from_pretrained builds its text tokenizer from `language_model_pretrained_name`. The only supported family is Qwen (detected by a case-insensitive 'qwen' substring in the repo name); any other name — Llama, Gemma, a local fine-tune directory without 'qwen' in the path — raises this ValueError before the audio processor is built.","triggerScenarios":"Passing a config or kwarg language_model_pretrained_name that does not contain 'qwen', e.g. 'meta-llama/Llama-3.1-8B', 'google/gemma-2-2b', or a local path like '/models/my-finetune' even when the underlying model is Qwen-based.","commonSituations":"Pointing at a locally saved copy of Qwen whose directory name dropped the 'qwen' token; attempting to swap the LM backbone to Llama/Gemma (the error message's sibling in vibevoice_processor.py even claims Llama/Gemma are supported — they are not wired up here); typos in the repo id.","solutions":["Use a Qwen-family checkpoint id such as 'Qwen/Qwen2.5-1.5B' (the default) for language_model_pretrained_name.","If using a local fine-tune, ensure the directory/path string still contains 'qwen' (case-insensitive), or rename it accordingly.","For other model families, subclass the processor and add a tokenizer branch instead of relying on this loader."],"exampleFix":"# before\nprocessor = VibeVoiceASRProcessor.from_pretrained('/checkpoints/my-asr-lm')  # no 'qwen' in path\n\n# after\nprocessor = VibeVoiceASRProcessor.from_pretrained(\n    ..., language_model_pretrained_name='Qwen/Qwen2.5-1.5B'\n)","handlingStrategy":"validation","validationCode":"name = config.get('language_model_pretrained_name') or 'Qwen/Qwen2.5-1.5B'\nif 'qwen' not in name.lower():\n    raise ValueError(f'ASR processor requires a Qwen tokenizer, got {name!r}')\nprocessor = VibeVoiceASRProcessor.from_pretrained(..., language_model_pretrained_name=name)","typeGuard":"def is_qwen_tokenizer_name(name: str) -> bool:\n    return isinstance(name, str) and 'qwen' in name.lower()","tryCatchPattern":"try:\n    processor = VibeVoiceASRProcessor.from_pretrained(model_path)\nexcept ValueError as e:\n    if 'Unsupported tokenizer type' in str(e):\n        processor = VibeVoiceASRProcessor.from_pretrained(\n            model_path, language_model_pretrained_name='Qwen/Qwen2.5-1.5B')\n    else:\n        raise","preventionTips":["Pin language_model_pretrained_name to a known Qwen repo id in your config.","Keep 'qwen' in local checkpoint directory names so the substring check matches.","Fail early with your own clearer message when the name is non-Qwen."],"tags":["tokenizer","config","asr","from-pretrained"],"backgroundTag":null,"analyzedSha":"94da20d98b2fa7688e9cbfaf7692ddb4954f7600","analyzedAt":"2026-08-15T04:12:07.418Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}