{"record":{"id":"ed44bf79f31fb19d","repo":"nodejs/node","slug":"payload-in-an-invalid-encoding","errorCode":null,"errorMessage":"payload in an invalid encoding","messagePattern":"payload in an invalid encoding","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"tools/gyp/pylib/packaging/metadata.py","lineNumber":230,"sourceCode":"\n    return urls\n\n\ndef _get_payload(msg: email.message.Message, source: Union[bytes, str]) -> str:\n    \"\"\"Get the body of the message.\"\"\"\n    # If our source is a str, then our caller has managed encodings for us,\n    # and we don't need to deal with it.\n    if isinstance(source, str):\n        payload: str = msg.get_payload()\n        return payload\n    # If our source is a bytes, then we're managing the encoding and we need\n    # to deal with it.\n    else:\n        bpayload: bytes = msg.get_payload(decode=True)\n        try:\n            return bpayload.decode(\"utf8\", \"strict\")\n        except UnicodeDecodeError:\n            raise ValueError(\"payload in an invalid encoding\")\n\n\n# The various parse_FORMAT functions here are intended to be as lenient as\n# possible in their parsing, while still returning a correctly typed\n# RawMetadata.\n#\n# To aid in this, we also generally want to do as little touching of the\n# data as possible, except where there are possibly some historic holdovers\n# that make valid data awkward to work with.\n#\n# While this is a lower level, intermediate format than our ``Metadata``\n# class, some light touch ups can make a massive difference in usability.\n\n# Map METADATA fields to RawMetadata.\n_EMAIL_TO_RAW_MAPPING = {\n    \"author\": \"author\",\n    \"author-email\": \"author_email\",\n    \"classifier\": \"classifiers\",","sourceCodeStart":212,"sourceCodeEnd":248,"githubUrl":"https://github.com/nodejs/node/blob/1b2de5e052fc0fb95fd7fb6846dcec4ade598e9e/tools/gyp/pylib/packaging/metadata.py#L212-L248","documentation":"Raised by packaging.metadata.parse_email when the metadata source is bytes and the email message body cannot be decoded as strict UTF-8. The library treats a str source as already encoding-managed by the caller, but for bytes it must decode the payload itself and rejects anything that is not valid UTF-8. It surfaces as a ValueError wrapping a UnicodeDecodeError.","triggerScenarios":"Calling Metadata.from_email(some_bytes) or parse_email(some_bytes) where the bytes payload contains non-UTF-8 sequences (e.g. Latin-1, CP1252, or a CTE like base64 that decodes to a non-UTF-8 charset). Only the bytes branch exercises decode('utf8','strict'); a str input never hits this path.","commonSituations":"Reading a METADATA/PKG-INFO file in binary mode from a sdist built on a non-UTF-8 locale, a corrupted download, or a file that declares a charset other than utf-8 in its MIME headers but whose decoded bytes are not UTF-8. Also seen when upstream packaging produced a file with a legacy 8-bit encoding.","solutions":["Decode the bytes yourself with the charset declared by the email message (msg.get_content_charset()) and pass a str to from_email so the library skips its internal decode.","If the file is genuinely UTF-8 but corrupted, re-fetch or rebuild the source distribution.","As a last resort, decode with errors='replace' to inspect the metadata, then fix the upstream producer."],"exampleFix":"# before\nwith open(path, 'rb') as f:\n    data = f.read()\nmeta = Metadata.from_email(data)\n\n# after\nwith open(path, 'r', encoding='utf-8') as f:\n    data = f.read()\nmeta = Metadata.from_email(data)","handlingStrategy":"validation","validationCode":"def safe_from_email(data: bytes):\n    import email\n    msg = email.message_from_bytes(data)\n    charset = msg.get_content_charset() or 'utf-8'\n    try:\n        text = data.decode(charset)\n    except (LookupError, UnicodeDecodeError):\n        text = data.decode('utf-8', errors='replace')\n    return Metadata.from_email(text)","typeGuard":"def is_utf8_decodable(b: bytes) -> bool:\n    try:\n        b.decode('utf-8', 'strict')\n        return True\n    except UnicodeDecodeError:\n        return False","tryCatchPattern":"try:\n    meta = Metadata.from_email(data)\nexcept ValueError as e:\n    if 'invalid encoding' in str(e):\n        # decode with declared charset and retry, or report\n        pass","preventionTips":["Prefer reading METADATA/PKG-INFO in text mode with utf-8 encoding.","If you must read bytes, decode using the email message's declared charset first.","Validate downloaded distributions are not truncated/corrupted before parsing."],"tags":["packaging","encoding","metadata","unicode"],"backgroundTag":null,"analyzedSha":"1b2de5e052fc0fb95fd7fb6846dcec4ade598e9e","analyzedAt":"2026-08-13T00:53:24.642Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}