{"record":{"id":"add54e59745111ce","repo":"pypa/pip","slug":"raw-is-too-large","errorCode":null,"errorMessage":"Raw is too large","messagePattern":"Raw is too large","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/msgpack/fallback.py","lineNumber":909,"sourceCode":"        raise ValueError(\"Dict is too large\")\n\n    def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT):\n        self._pack_map_header(n)\n        for k, v in pairs:\n            self._pack(k, nest_limit - 1)\n            self._pack(v, nest_limit - 1)\n\n    def _pack_raw_header(self, n):\n        if n <= 0x1F:\n            self._buffer.write(struct.pack(\"B\", 0xA0 + n))\n        elif self._use_bin_type and n <= 0xFF:\n            self._buffer.write(struct.pack(\">BB\", 0xD9, n))\n        elif n <= 0xFFFF:\n            self._buffer.write(struct.pack(\">BH\", 0xDA, n))\n        elif n <= 0xFFFFFFFF:\n            self._buffer.write(struct.pack(\">BI\", 0xDB, n))\n        else:\n            raise ValueError(\"Raw is too large\")\n\n    def _pack_bin_header(self, n):\n        if not self._use_bin_type:\n            return self._pack_raw_header(n)\n        elif n <= 0xFF:\n            return self._buffer.write(struct.pack(\">BB\", 0xC4, n))\n        elif n <= 0xFFFF:\n            return self._buffer.write(struct.pack(\">BH\", 0xC5, n))\n        elif n <= 0xFFFFFFFF:\n            return self._buffer.write(struct.pack(\">BI\", 0xC6, n))\n        else:\n            raise ValueError(\"Bin is too large\")\n\n    def bytes(self):\n        \"\"\"Return internal buffer contents as bytes object\"\"\"\n        return self._buffer.getvalue()\n\n    def reset(self):","sourceCodeStart":891,"sourceCodeEnd":927,"githubUrl":"https://github.com/pypa/pip/blob/f399c3718970b1b0e2478dac5296eb62679a9b86/src/pip/_vendor/msgpack/fallback.py#L891-L927","documentation":"Raised by msgpack's pure-Python Packer._pack_raw_header when the raw string length exceeds 0xFFFFFFFF (4,294,967,295 bytes). The raw/str format encodes length in a 32-bit field, so strings whose UTF-8 encoding exceeds ~4 GiB cannot be represented. This is the string/raw-type-specific size limit, distinct from the bin type limit.","triggerScenarios":"Packing a str whose UTF-8 encoded byte length exceeds 0xFFFFFFFF. Hit via packer.pack(very_long_string) or packb(very_long_string). Also reached internally when _pack_bin_header falls through to _pack_raw_header in non-bin-type mode.","commonSituations":"Serializing very large text payloads (logs, JSON, base64 data, generated source code) as a single msgpack string; embedding large file contents as strings.","solutions":["Encode the string to bytes and chunk into multiple bin messages each under 4 GiB.","Store large text externally and pack only a reference.","Pre-check len(s.encode('utf-8')) before packing.","Use a streaming approach for very large text data."],"exampleFix":"# before\npackb(huge_string)  # raises if UTF-8 encoding > 4 GiB\n\n# after — encode and chunk\nencoded = huge_string.encode('utf-8')\nMAX = 0xFFFFFFFF\nchunks = [encoded[i:i+MAX] for i in range(0, len(encoded), MAX)]\npackb([c for c in chunks])  # reassemble on unpack","handlingStrategy":"validation","validationCode":"MSGPACK_MAX_STR = 0xFFFFFFFF\n\ndef is_serializable_string(s: str) -> bool:\n    return len(s.encode('utf-8')) < MSGPACK_MAX_STR","typeGuard":"def is_serializable_str(obj) -> bool:\n    return isinstance(obj, str) and len(obj.encode('utf-8')) < 0xFFFFFFFF","tryCatchPattern":"try:\n    packed = packer.pack(huge_string)\nexcept ValueError as e:\n    if 'too large' in str(e).lower():\n        encoded = huge_string.encode('utf-8')\n        chunks = [encoded[i:i+0xFFFFFFFF] for i in range(0, len(encoded), 0xFFFFFFFF)]\n        packed = packer.pack([c for c in chunks])\n    else:\n        raise","preventionTips":["Check UTF-8 encoded length for large strings before packing.","Store very large text externally and pack references.","Chunk large text payloads."],"tags":["msgpack","serialization","size-limit","string"],"backgroundTag":null,"analyzedSha":"f399c3718970b1b0e2478dac5296eb62679a9b86","analyzedAt":"2026-08-08T23:01:42.227Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}