{"record":{"id":"f81a2ed0fcf14164","repo":"microsoft/VibeVoice","slug":"please-install-huggingface-hub-pip-install-huggin","errorCode":null,"errorMessage":"Please install huggingface_hub: pip install huggingface_hub","messagePattern":"Please install huggingface_hub: pip install huggingface_hub","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"warning","filePath":"vllm_plugin/tools/generate_tokenizer_files.py","lineNumber":156,"sourceCode":"            {{- '<|im_end|>\\\\n' }}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\\\n' }}\n{%- endif %}\"\"\"\n\n\n# Default to Qwen2.5-7B which has all the extended tokens (151646-151664)\nDEFAULT_QWEN_MODEL = \"Qwen/Qwen2.5-7B\"\n\n\ndef download_qwen_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWEN_MODEL) -> None:\n    \"\"\"Download base tokenizer files from Qwen2.5 (which includes extended tokens).\"\"\"\n    try:\n        from huggingface_hub import hf_hub_download\n    except ImportError:\n        raise ImportError(\"Please install huggingface_hub: pip install huggingface_hub\")\n    \n    files_to_download = [\n        \"vocab.json\",\n        \"merges.txt\",\n        \"tokenizer.json\",\n        \"tokenizer_config.json\",\n    ]\n    \n    os.makedirs(output_dir, exist_ok=True)\n    \n    for filename in files_to_download:\n        print(f\"Downloading {filename} from {qwen_model}...\")\n        hf_hub_download(\n            repo_id=qwen_model,\n            filename=filename,\n            local_dir=output_dir,\n            local_dir_use_symlinks=False,\n        )","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/microsoft/VibeVoice/blob/94da20d98b2fa7688e9cbfaf7692ddb4954f7600/vllm_plugin/tools/generate_tokenizer_files.py#L138-L174","documentation":"Raised by tools/generate_tokenizer_files.py when generating the merged VibeVoice tokenizer: it lazily imports hf_hub_download from huggingface_hub inside download_qwen_tokenizer_files() to fetch Qwen2.5 tokenizer files (vocab.json, merges.txt, tokenizer.json, tokenizer_config.json), and ImportError is converted to this actionable message when the package is absent.","triggerScenarios":"Running the tokenizer-generation tool in a minimal venv that has vllm/torch but not huggingface_hub; environments where huggingface_hub was uninstalled as a 'transitive' dependency; offline pip installs that skipped optional extras.","commonSituations":"CI images trimmed to reduce size; Dockerfiles installing only runtime deps but invoking build tools; dependency-resolver conflicts that removed huggingface_hub after a pip install --force of another package.","solutions":["pip install huggingface_hub (or add it to the tool's requirements) and rerun the script.","If offline, pre-download the four files from the Qwen/Qwen2.5-7B repo manually and place them in the output dir, skipping the download step.","Pin a compatible version if a resolver fight occurs: pip install 'huggingface_hub>=0.20'.","Verify import works: python -c 'from huggingface_hub import hf_hub_download; print(\"ok\")'."],"exampleFix":"# before\n$ python -m vllm_plugin.tools.generate_tokenizer_files --output-dir ./tok\nImportError: Please install huggingface_hub: pip install huggingface_hub\n\n# after\n$ pip install huggingface_hub\n$ python -m vllm_plugin.tools.generate_tokenizer_files --output-dir ./tok","handlingStrategy":"validation","validationCode":"def ensure_hf_hub():\n    try:\n        import huggingface_hub  # noqa: F401\n        return True\n    except ImportError:\n        return False\n\nif not ensure_hf_hub():\n    raise SystemExit(\"pip install huggingface_hub before running this tool\")","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Add huggingface_hub to the tool's requirements file or the CI image.","For offline environments, pre-download the four Qwen tokenizer files into the output dir.","Smoke-test `python -c \"from huggingface_hub import hf_hub_download\"` in Dockerfile builds."],"tags":["dependencies","import-error","huggingface","tokenizer","tooling"],"backgroundTag":null,"analyzedSha":"94da20d98b2fa7688e9cbfaf7692ddb4954f7600","analyzedAt":"2026-08-15T04:12:07.418Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}