pypa/pip · error · ValueError

Array is too large

Error message

Array is too large

What it means

Raised by msgpack's pure-Python Packer._pack_array_header when the array element count exceeds 0xFFFFFFFF (4,294,967,295). The msgpack array header uses at most a 32-bit length field, so arrays with more than ~4.29 billion elements cannot be encoded. This is hit when packing a list or tuple that is too large.

Source

Thrown at src/pip/_vendor/msgpack/fallback.py:882

        elif L <= 0xFFFF:
            self._buffer.write(b"\xc8" + struct.pack(">H", L))
        else:
            self._buffer.write(b"\xc9" + struct.pack(">I", L))
        self._buffer.write(struct.pack("B", typecode))
        self._buffer.write(data)
        if self._autoreset:
            ret = self._buffer.getvalue()
            self._buffer = BytesIO()
            return ret

    def _pack_array_header(self, n):
        if n <= 0x0F:
            return self._buffer.write(struct.pack("B", 0x90 + n))
        if n <= 0xFFFF:
            return self._buffer.write(struct.pack(">BH", 0xDC, n))
        if n <= 0xFFFFFFFF:
            return self._buffer.write(struct.pack(">BI", 0xDD, n))
        raise ValueError("Array is too large")

    def _pack_map_header(self, n):
        if n <= 0x0F:
            return self._buffer.write(struct.pack("B", 0x80 + n))
        if n <= 0xFFFF:
            return self._buffer.write(struct.pack(">BH", 0xDE, n))
        if n <= 0xFFFFFFFF:
            return self._buffer.write(struct.pack(">BI", 0xDF, n))
        raise ValueError("Dict is too large")

    def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT):
        self._pack_map_header(n)
        for k, v in pairs:
            self._pack(k, nest_limit - 1)
            self._pack(v, nest_limit - 1)

    def _pack_raw_header(self, n):
        if n <= 0x1F:

View on GitHub (pinned to f399c37189)

Solutions

1. Split the array into multiple messages each under 4.29 billion elements (practically, chunk far smaller).
2. Use msgpack streaming (Packer in a loop, feeding to a stream) instead of one giant array.
3. Filter or paginate the collection before serialization.
4. Consider a columnar format (Parquet, Arrow) for very large tabular data instead of msgpack.

Example fix

# before
packb(huge_list)  # raises if len > 0xFFFFFFFF

# after — stream chunks
packer = msgpack.Packer()
for chunk in chunks_of(huge_list, batch_size=1_000_000):
    stream.write(packer.pack(chunk))

MSGPACK_MAX_ARRAY = 0xFFFFFFFF

def is_serializable_array(arr) -> bool:
    return len(arr) < MSGPACK_MAX_ARRAY

def is_serializable_list(obj) -> bool:
    return isinstance(obj, (list, tuple)) and len(obj) < 0xFFFFFFFF

try:
    packed = packer.pack(huge_list)
except ValueError as e:
    if 'too large' in str(e).lower():
        for batch in batched(huge_list, 1_000_000):
            stream.write(packer.pack(list(batch)))
    else:
        raise

When it happens

Trigger: Packing a list or tuple with more than 0xFFFFFFFF elements via packer.pack(obj) or packb(obj). Encountered when serializing very large in-memory collections.

Common situations: Serializing massive datasets (logs, event streams, ML feature vectors) as a single flat array; accumulating unbounded data before serialization; batch processing without size limits.

Related errors

• Memoryview is too large
• Dict is too large
• Raw is too large
• Cannot serialize {obj!r} where tzinfo=None
• Cannot serialize {obj!r}

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/3a132ad361c7ee07. Report an issue: GitHub.