huggingface/tokenizers · error · ImportError

We couldn't import IPython utils for html display. Are you…

Error message

We couldn't import IPython utils for html display.
Are you running in a notebook?
You can also pass `default_to_notebook=False` to get back raw HTML.

What it means

EncodingVisualizer.__init__ lazily imports IPython's display utilities so it can render tokenizations as HTML in notebooks. When IPython (or its nbformat sub-dependency) is not installed and default_to_notebook was not explicitly set to False, the ImportError is re-raised with this explanatory message.

Solutions

1. Install IPython: pip install ipython (nbformat is pulled in with tokenizers' extras or install it too).
2. Pass `default_to_notebook=False` to EncodingVisualizer so it returns raw HTML instead of importing IPython.
3. If you only need the HTML string, call `to_html()`/inspect output after constructing with default_to_notebook=False.

Example fix

// before
viz = EncodingVisualizer(tokenizer)  # ImportError outside notebooks
// after
viz = EncodingVisualizer(tokenizer, default_to_notebook=False)
html = viz.to_html(text)  # raw HTML string, no IPython needed
// or: pip install ipython

try:
    import IPython.display  # noqa: F401
    HAS_IPYTHON = True
except ImportError:
    HAS_IPYTHON = False

viz_kwargs = {} if HAS_IPYTHON else {"default_to_notebook": False}
viz = EncodingVisualizer(tokenizer, **viz_kwargs)

def ipython_available() -> bool:
    try:
        import IPython.display
        return True
    except ImportError:
        return False

try:
    viz = EncodingVisualizer(tokenizer)
except ImportError:
    viz = EncodingVisualizer(tokenizer, default_to_notebook=False)

When it happens

Trigger: Constructing `EncodingVisualizer(tokenizer)` (default default_to_notebook=True) in an environment where `from IPython.display import HTML, display` or nbformat import fails — typically IPython is not installed.

Common situations: Using the visualizer in a plain script, production service, or bare virtualenv without IPython; slim Docker images that omit notebook tooling; relying on the visualizer outside Jupyter without disabling notebook mode.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

• encode: `sequence` can't be `None`
• encode_batch: `inputs` can't be `None`
• async_encode_batch: `inputs` can't be `None`
• async_encode_batch_fast: `inputs` can't be `None`
• None input is not valid. Should be a list of integers.

AI-assisted analysis of huggingface/tokenizers@6cfd9d385c (2026-09-09). Data as JSON: /api/errors/c6cef41cbb3fb86d. Report an issue: GitHub.

Appendix: source

Thrown at bindings/python/py_src/tokenizers/tools/visualizer.py:104

    def __init__(
        self,
        tokenizer: Tokenizer,
        default_to_notebook: bool = True,
        annotation_converter: Optional[Callable[[Any], Annotation]] = None,
    ):
        if default_to_notebook:
            try:
                from IPython.display import HTML, display  # type: ignore[attr-defined]
            except ImportError:
                try:
                    from IPython.core.display import HTML, display  # type: ignore[attr-defined]
                except ImportError:
                    msg = (
                        "We couldn't import IPython utils for html display.\n"
                        "Are you running in a notebook?\n"
                        "You can also pass `default_to_notebook=False` to get back raw HTML.\n"
                    )
                    raise ImportError(msg) from None
        self.tokenizer = tokenizer
        self.default_to_notebook = default_to_notebook
        self.annotation_coverter = annotation_converter
        pass

    def __call__(
        self,
        text: str,
        annotations: Optional[List[Any]] = None,
        default_to_notebook: Optional[bool] = None,
    ) -> Optional[str]:
        """
        Build a visualization of the given text

        Args:
            text (:obj:`str`):
                The text to tokenize

View on GitHub (pinned to 6cfd9d385c)