{"record":{"id":"1cd08be5241a5824","repo":"pypa/pip","slug":"dict-is-too-large","errorCode":null,"errorMessage":"Dict is too large","messagePattern":"Dict is too large","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/msgpack/fallback.py","lineNumber":891,"sourceCode":"            return ret\n\n    def _pack_array_header(self, n):\n        if n <= 0x0F:\n            return self._buffer.write(struct.pack(\"B\", 0x90 + n))\n        if n <= 0xFFFF:\n            return self._buffer.write(struct.pack(\">BH\", 0xDC, n))\n        if n <= 0xFFFFFFFF:\n            return self._buffer.write(struct.pack(\">BI\", 0xDD, n))\n        raise ValueError(\"Array is too large\")\n\n    def _pack_map_header(self, n):\n        if n <= 0x0F:\n            return self._buffer.write(struct.pack(\"B\", 0x80 + n))\n        if n <= 0xFFFF:\n            return self._buffer.write(struct.pack(\">BH\", 0xDE, n))\n        if n <= 0xFFFFFFFF:\n            return self._buffer.write(struct.pack(\">BI\", 0xDF, n))\n        raise ValueError(\"Dict is too large\")\n\n    def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT):\n        self._pack_map_header(n)\n        for k, v in pairs:\n            self._pack(k, nest_limit - 1)\n            self._pack(v, nest_limit - 1)\n\n    def _pack_raw_header(self, n):\n        if n <= 0x1F:\n            self._buffer.write(struct.pack(\"B\", 0xA0 + n))\n        elif self._use_bin_type and n <= 0xFF:\n            self._buffer.write(struct.pack(\">BB\", 0xD9, n))\n        elif n <= 0xFFFF:\n            self._buffer.write(struct.pack(\">BH\", 0xDA, n))\n        elif n <= 0xFFFFFFFF:\n            self._buffer.write(struct.pack(\">BI\", 0xDB, n))\n        else:\n            raise ValueError(\"Raw is too large\")","sourceCodeStart":873,"sourceCodeEnd":909,"githubUrl":"https://github.com/pypa/pip/blob/f399c3718970b1b0e2478dac5296eb62679a9b86/src/pip/_vendor/msgpack/fallback.py#L873-L909","documentation":"Raised by msgpack's pure-Python Packer._pack_map_header when the dictionary key-value pair count exceeds 0xFFFFFFFF (4,294,967,295). The msgpack map header uses at most a 32-bit length field, so maps with more than ~4.29 billion entries cannot be encoded.","triggerScenarios":"Packing a dict with more than 0xFFFFFFFF entries via packer.pack(obj) or packb(obj). Also raised via pack_map_pairs or pack_map_header when the count is oversized.","commonSituations":"Serializing a massive key-value store, cache dump, or index as a single msgpack map; unbounded accumulation of entries; in-memory aggregation without eviction.","solutions":["Split the map into multiple messages each under the limit.","Stream key-value pairs using msgpack's streaming Packer rather than building one map.","Use a database or external key-value store for very large mappings and serialize only a reference.","Pre-check len(d) and reject/batch if it exceeds the limit."],"exampleFix":"# before\npackb(huge_dict)  # raises if len > 0xFFFFFFFF\n\n# after — batch into smaller maps\nbatch_size = 1_000_000\nitems = list(huge_dict.items())\nfor i in range(0, len(items), batch_size):\n    batch = dict(items[i:i+batch_size])\n    stream.write(msgpack.packb(batch))","handlingStrategy":"validation","validationCode":"MSGPACK_MAX_MAP = 0xFFFFFFFF\n\ndef is_serializable_dict(d) -> bool:\n    return len(d) < MSGPACK_MAX_MAP","typeGuard":"def is_serializable_dict(obj) -> bool:\n    return isinstance(obj, dict) and len(obj) < 0xFFFFFFFF","tryCatchPattern":"try:\n    packed = packer.pack(huge_dict)\nexcept ValueError as e:\n    if 'too large' in str(e).lower():\n        items = list(huge_dict.items())\n        for i in range(0, len(items), 1_000_000):\n            stream.write(packer.pack(dict(items[i:i+1_000_000])))\n    else:\n        raise","preventionTips":["Set practical map size limits.","Use external key-value stores for very large mappings.","Batch large dicts before serialization."],"tags":["msgpack","serialization","size-limit","dict"],"backgroundTag":null,"analyzedSha":"f399c3718970b1b0e2478dac5296eb62679a9b86","analyzedAt":"2026-08-08T23:01:42.227Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}