pypa/pip · error · ImportError

To enable chardet encoding guessing, please install the char

Error message

To enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/

What it means

Raised by Lexer._preprocess_lexer_input when the lexer's encoding is set to 'chardet' but the chardet library is not available. In upstream pygments this triggers chardet.detect for bytes input; in pip's vendored copy chardet is deliberately NOT vendored, so an ImportError is forced and re-raised with this message. Any bytes input under encoding='chardet' will hit it.

Source

Thrown at src/pip/_vendor/pygments/lexer.py:218

        that it works like a static method (no ``self`` or ``cls``
        parameter) and the return value is automatically converted to
        `float`. If the return value is an object that is boolean `False`
        it's the same as if the return values was ``0.0``.
        """

    def _preprocess_lexer_input(self, text):
        """Apply preprocessing such as decoding the input, removing BOM and normalizing newlines."""

        if not isinstance(text, str):
            if self.encoding == 'guess':
                text, _ = guess_decode(text)
            elif self.encoding == 'chardet':
                try:
                    # pip vendoring note: this code is not reachable by pip,
                    # removed import of chardet to make it clear.
                    raise ImportError('chardet is not vendored by pip')
                except ImportError as e:
                    raise ImportError('To enable chardet encoding guessing, '
                                      'please install the chardet library '
                                      'from http://chardet.feedparser.org/') from e
                # check for BOM first
                decoded = None
                for bom, encoding in _encoding_map:
                    if text.startswith(bom):
                        decoded = text[len(bom):].decode(encoding, 'replace')
                        break
                # no BOM found, so use chardet
                if decoded is None:
                    enc = chardet.detect(text[:1024])  # Guess using first 1KB
                    decoded = text.decode(enc.get('encoding') or 'utf-8',
                                          'replace')
                text = decoded
            else:
                text = text.decode(self.encoding)
                if text.startswith('\ufeff'):
                    text = text[len('\ufeff'):]

View on GitHub (pinned to f399c37189)

Solutions

Decode bytes to str before passing to the lexer (text = data.decode('utf-8', 'replace')).
Use encoding='guess' (which uses the BOM heuristic) or an explicit encoding instead of 'chardet' when on the vendored copy.
Use a standalone (non-vendored) pygments with chardet installed if you truly need chardet detection.
Avoid importing pip's vendored pygments for application code; depend on pygments directly.

Example fix

# before
lexer = PythonLexer(encoding='chardet')
lexer.get_tokens(raw_bytes)
# after
lexer = PythonLexer()
lexer.get_tokens(raw_bytes.decode('utf-8', 'replace'))

Defensive patterns

Strategy: validation

Validate before calling

if isinstance(code, bytes):
    code = code.decode('utf-8', 'replace')  # avoid chardet path entirely

Type guard

def is_str_input(code) -> bool:
    return isinstance(code, str)

Try / catch

try:
    tokens = lexer.get_tokens(data)
except ImportError as e:
    if 'chardet' in str(e):
        data = data.decode('utf-8', 'replace')
        tokens = lexer.get_tokens(data)
    else:
        raise

Prevention

Always decode bytes to str before lexing.
Do not rely on pip's vendored pygments for chardet; use standalone pygments with chardet installed if needed.
Avoid encoding='chardet' on vendored copies.

When it happens

Trigger: Constructing a lexer with encoding='chardet' and feeding it bytes (not str) input, in the pip-vendored pygments. The branch at lexer.py:212-220 forces ImportError because pip does not vendor chardet.

Common situations: Using pip's vendored pygments directly (importing from pip._vendor.pygments) instead of a standalone pygments install; passing file bytes to a lexer configured for chardet detection.

Related errors

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/ca854064b11090b8. Report an issue: GitHub.