microsoft/VibeVoice · warning · ImportError

Please install huggingface_hub: pip install huggingface_hub

Error message

Please install huggingface_hub: pip install huggingface_hub

What it means

Raised by tools/generate_tokenizer_files.py when generating the merged VibeVoice tokenizer: it lazily imports hf_hub_download from huggingface_hub inside download_qwen_tokenizer_files() to fetch Qwen2.5 tokenizer files (vocab.json, merges.txt, tokenizer.json, tokenizer_config.json), and ImportError is converted to this actionable message when the package is absent.

Source

Thrown at vllm_plugin/tools/generate_tokenizer_files.py:156

            {{- '<|im_end|>\\n' }}
        {%- endif %}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\\n' }}
{%- endif %}"""


# Default to Qwen2.5-7B which has all the extended tokens (151646-151664)
DEFAULT_QWEN_MODEL = "Qwen/Qwen2.5-7B"


def download_qwen_tokenizer_files(output_dir: str, qwen_model: str = DEFAULT_QWEN_MODEL) -> None:
    """Download base tokenizer files from Qwen2.5 (which includes extended tokens)."""
    try:
        from huggingface_hub import hf_hub_download
    except ImportError:
        raise ImportError("Please install huggingface_hub: pip install huggingface_hub")
    
    files_to_download = [
        "vocab.json",
        "merges.txt",
        "tokenizer.json",
        "tokenizer_config.json",
    ]
    
    os.makedirs(output_dir, exist_ok=True)
    
    for filename in files_to_download:
        print(f"Downloading {filename} from {qwen_model}...")
        hf_hub_download(
            repo_id=qwen_model,
            filename=filename,
            local_dir=output_dir,
            local_dir_use_symlinks=False,
        )

View on GitHub (pinned to 94da20d98b)

Solutions

1. pip install huggingface_hub (or add it to the tool's requirements) and rerun the script.
2. If offline, pre-download the four files from the Qwen/Qwen2.5-7B repo manually and place them in the output dir, skipping the download step.
3. Pin a compatible version if a resolver fight occurs: pip install 'huggingface_hub>=0.20'.
4. Verify import works: python -c 'from huggingface_hub import hf_hub_download; print("ok")'.

Example fix

# before
$ python -m vllm_plugin.tools.generate_tokenizer_files --output-dir ./tok
ImportError: Please install huggingface_hub: pip install huggingface_hub

# after
$ pip install huggingface_hub
$ python -m vllm_plugin.tools.generate_tokenizer_files --output-dir ./tok

def ensure_hf_hub():
    try:
        import huggingface_hub  # noqa: F401
        return True
    except ImportError:
        return False

if not ensure_hf_hub():
    raise SystemExit("pip install huggingface_hub before running this tool")

When it happens

Trigger: Running the tokenizer-generation tool in a minimal venv that has vllm/torch but not huggingface_hub; environments where huggingface_hub was uninstalled as a 'transitive' dependency; offline pip installs that skipped optional extras.

Common situations: CI images trimmed to reduce size; Dockerfiles installing only runtime deps but invoking build tools; dependency-resolver conflicts that removed huggingface_hub after a pip install --force of another package.

Related errors

• GroupNorm doesn't support causal evaluation.
• Unsupported mixer layer: {mixer_layer}
• Unsupported norm type: {layernorm}
• Unsupported dist_type: {dist_type}, expected 'fix' or 'gauss
• Unsupported tokenizer type for {language_model_pretrained_na

AI-assisted analysis of microsoft/VibeVoice@94da20d98b (2026-08-15). Data as JSON: /api/errors/f81a2ed0fcf14164. Report an issue: GitHub.