{"record":{"id":"ca854064b11090b8","repo":"pypa/pip","slug":"to-enable-chardet-encoding-guessing-please-instal","errorCode":null,"errorMessage":"To enable chardet encoding guessing, please install the chardet library from http://chardet.feedparser.org/","messagePattern":"To enable chardet encoding guessing, please install the chardet library from http://chardet\\.feedparser\\.org/","errorType":"exception","errorClass":"ImportError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/pygments/lexer.py","lineNumber":218,"sourceCode":"        that it works like a static method (no ``self`` or ``cls``\n        parameter) and the return value is automatically converted to\n        `float`. If the return value is an object that is boolean `False`\n        it's the same as if the return values was ``0.0``.\n        \"\"\"\n\n    def _preprocess_lexer_input(self, text):\n        \"\"\"Apply preprocessing such as decoding the input, removing BOM and normalizing newlines.\"\"\"\n\n        if not isinstance(text, str):\n            if self.encoding == 'guess':\n                text, _ = guess_decode(text)\n            elif self.encoding == 'chardet':\n                try:\n                    # pip vendoring note: this code is not reachable by pip,\n                    # removed import of chardet to make it clear.\n                    raise ImportError('chardet is not vendored by pip')\n                except ImportError as e:\n                    raise ImportError('To enable chardet encoding guessing, '\n                                      'please install the chardet library '\n                                      'from http://chardet.feedparser.org/') from e\n                # check for BOM first\n                decoded = None\n                for bom, encoding in _encoding_map:\n                    if text.startswith(bom):\n                        decoded = text[len(bom):].decode(encoding, 'replace')\n                        break\n                # no BOM found, so use chardet\n                if decoded is None:\n                    enc = chardet.detect(text[:1024])  # Guess using first 1KB\n                    decoded = text.decode(enc.get('encoding') or 'utf-8',\n                                          'replace')\n                text = decoded\n            else:\n                text = text.decode(self.encoding)\n                if text.startswith('\\ufeff'):\n                    text = text[len('\\ufeff'):]","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/pypa/pip/blob/f399c3718970b1b0e2478dac5296eb62679a9b86/src/pip/_vendor/pygments/lexer.py#L200-L236","documentation":"Raised by Lexer._preprocess_lexer_input when the lexer's encoding is set to 'chardet' but the chardet library is not available. In upstream pygments this triggers chardet.detect for bytes input; in pip's vendored copy chardet is deliberately NOT vendored, so an ImportError is forced and re-raised with this message. Any bytes input under encoding='chardet' will hit it.","triggerScenarios":"Constructing a lexer with encoding='chardet' and feeding it bytes (not str) input, in the pip-vendored pygments. The branch at lexer.py:212-220 forces ImportError because pip does not vendor chardet.","commonSituations":"Using pip's vendored pygments directly (importing from pip._vendor.pygments) instead of a standalone pygments install; passing file bytes to a lexer configured for chardet detection.","solutions":["Decode bytes to str before passing to the lexer (text = data.decode('utf-8', 'replace')).","Use encoding='guess' (which uses the BOM heuristic) or an explicit encoding instead of 'chardet' when on the vendored copy.","Use a standalone (non-vendored) pygments with chardet installed if you truly need chardet detection.","Avoid importing pip's vendored pygments for application code; depend on pygments directly."],"exampleFix":"# before\nlexer = PythonLexer(encoding='chardet')\nlexer.get_tokens(raw_bytes)\n# after\nlexer = PythonLexer()\nlexer.get_tokens(raw_bytes.decode('utf-8', 'replace'))","handlingStrategy":"validation","validationCode":"if isinstance(code, bytes):\n    code = code.decode('utf-8', 'replace')  # avoid chardet path entirely","typeGuard":"def is_str_input(code) -> bool:\n    return isinstance(code, str)","tryCatchPattern":"try:\n    tokens = lexer.get_tokens(data)\nexcept ImportError as e:\n    if 'chardet' in str(e):\n        data = data.decode('utf-8', 'replace')\n        tokens = lexer.get_tokens(data)\n    else:\n        raise","preventionTips":["Always decode bytes to str before lexing.","Do not rely on pip's vendored pygments for chardet; use standalone pygments with chardet installed if needed.","Avoid encoding='chardet' on vendored copies."],"tags":["python","pygments","lexer","encoding","vendored"],"backgroundTag":null,"analyzedSha":"f399c3718970b1b0e2478dac5296eb62679a9b86","analyzedAt":"2026-08-08T23:01:42.227Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}