{"record":{"id":"3a132ad361c7ee07","repo":"pypa/pip","slug":"array-is-too-large","errorCode":null,"errorMessage":"Array is too large","messagePattern":"Array is too large","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/msgpack/fallback.py","lineNumber":882,"sourceCode":"        elif L <= 0xFFFF:\n            self._buffer.write(b\"\\xc8\" + struct.pack(\">H\", L))\n        else:\n            self._buffer.write(b\"\\xc9\" + struct.pack(\">I\", L))\n        self._buffer.write(struct.pack(\"B\", typecode))\n        self._buffer.write(data)\n        if self._autoreset:\n            ret = self._buffer.getvalue()\n            self._buffer = BytesIO()\n            return ret\n\n    def _pack_array_header(self, n):\n        if n <= 0x0F:\n            return self._buffer.write(struct.pack(\"B\", 0x90 + n))\n        if n <= 0xFFFF:\n            return self._buffer.write(struct.pack(\">BH\", 0xDC, n))\n        if n <= 0xFFFFFFFF:\n            return self._buffer.write(struct.pack(\">BI\", 0xDD, n))\n        raise ValueError(\"Array is too large\")\n\n    def _pack_map_header(self, n):\n        if n <= 0x0F:\n            return self._buffer.write(struct.pack(\"B\", 0x80 + n))\n        if n <= 0xFFFF:\n            return self._buffer.write(struct.pack(\">BH\", 0xDE, n))\n        if n <= 0xFFFFFFFF:\n            return self._buffer.write(struct.pack(\">BI\", 0xDF, n))\n        raise ValueError(\"Dict is too large\")\n\n    def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT):\n        self._pack_map_header(n)\n        for k, v in pairs:\n            self._pack(k, nest_limit - 1)\n            self._pack(v, nest_limit - 1)\n\n    def _pack_raw_header(self, n):\n        if n <= 0x1F:","sourceCodeStart":864,"sourceCodeEnd":900,"githubUrl":"https://github.com/pypa/pip/blob/f399c3718970b1b0e2478dac5296eb62679a9b86/src/pip/_vendor/msgpack/fallback.py#L864-L900","documentation":"Raised by msgpack's pure-Python Packer._pack_array_header when the array element count exceeds 0xFFFFFFFF (4,294,967,295). The msgpack array header uses at most a 32-bit length field, so arrays with more than ~4.29 billion elements cannot be encoded. This is hit when packing a list or tuple that is too large.","triggerScenarios":"Packing a list or tuple with more than 0xFFFFFFFF elements via packer.pack(obj) or packb(obj). Encountered when serializing very large in-memory collections.","commonSituations":"Serializing massive datasets (logs, event streams, ML feature vectors) as a single flat array; accumulating unbounded data before serialization; batch processing without size limits.","solutions":["Split the array into multiple messages each under 4.29 billion elements (practically, chunk far smaller).","Use msgpack streaming (Packer in a loop, feeding to a stream) instead of one giant array.","Filter or paginate the collection before serialization.","Consider a columnar format (Parquet, Arrow) for very large tabular data instead of msgpack."],"exampleFix":"# before\npackb(huge_list)  # raises if len > 0xFFFFFFFF\n\n# after — stream chunks\npacker = msgpack.Packer()\nfor chunk in chunks_of(huge_list, batch_size=1_000_000):\n    stream.write(packer.pack(chunk))","handlingStrategy":"validation","validationCode":"MSGPACK_MAX_ARRAY = 0xFFFFFFFF\n\ndef is_serializable_array(arr) -> bool:\n    return len(arr) < MSGPACK_MAX_ARRAY","typeGuard":"def is_serializable_list(obj) -> bool:\n    return isinstance(obj, (list, tuple)) and len(obj) < 0xFFFFFFFF","tryCatchPattern":"try:\n    packed = packer.pack(huge_list)\nexcept ValueError as e:\n    if 'too large' in str(e).lower():\n        for batch in batched(huge_list, 1_000_000):\n            stream.write(packer.pack(list(batch)))\n    else:\n        raise","preventionTips":["Stream large collections instead of packing as one array.","Set practical batch size limits far below 4 billion.","Use streaming msgpack for large datasets."],"tags":["msgpack","serialization","size-limit","array"],"backgroundTag":null,"analyzedSha":"f399c3718970b1b0e2478dac5296eb62679a9b86","analyzedAt":"2026-08-08T23:01:42.227Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}