pypa/pip · error · ValueError

Too large data

Error message

Too large data

What it means

Raised by msgpack's pure-Python Packer.pack_ext_type when the data payload length exceeds 0xFFFFFFFF (4 GiB − 1). The msgpack ext format encodes the data length in a 32-bit field, so the maximum single ext payload is just under 4 GiB. This is the ext-type-specific equivalent of the bin/string size limits.

Source

Thrown at src/pip/_vendor/msgpack/fallback.py:851

    def pack_map_header(self, n):
        if n >= 2**32:
            raise ValueError
        self._pack_map_header(n)
        if self._autoreset:
            ret = self._buffer.getvalue()
            self._buffer = BytesIO()
            return ret

    def pack_ext_type(self, typecode, data):
        if not isinstance(typecode, int):
            raise TypeError("typecode must have int type.")
        if not 0 <= typecode <= 127:
            raise ValueError("typecode should be 0-127")
        if not isinstance(data, bytes):
            raise TypeError("data must have bytes type")
        L = len(data)
        if L > 0xFFFFFFFF:
            raise ValueError("Too large data")
        if L == 1:
            self._buffer.write(b"\xd4")
        elif L == 2:
            self._buffer.write(b"\xd5")
        elif L == 4:
            self._buffer.write(b"\xd6")
        elif L == 8:
            self._buffer.write(b"\xd7")
        elif L == 16:
            self._buffer.write(b"\xd8")
        elif L <= 0xFF:
            self._buffer.write(b"\xc7" + struct.pack("B", L))
        elif L <= 0xFFFF:
            self._buffer.write(b"\xc8" + struct.pack(">H", L))
        else:
            self._buffer.write(b"\xc9" + struct.pack(">I", L))
        self._buffer.write(struct.pack("B", typecode))
        self._buffer.write(data)

View on GitHub (pinned to f399c37189)

Solutions

1. Split the payload into multiple ext messages each under 4 GiB and reassemble on unpack.
2. Store the large payload externally and pack only a reference/URL in the ext data.
3. Use the standard bin type with chunking instead of a custom ext type for very large blobs.
4. Pre-check len(data) <= 0xFFFFFFFF before calling pack_ext_type.

Example fix

# before
packer.pack_ext_type(1, huge_bytes)  # raises if > 4 GiB

# after — chunk the payload
MAX = 0xFFFFFFFF
chunks = [huge_bytes[i:i+MAX] for i in range(0, len(huge_bytes), MAX)]
for i, chunk in enumerate(chunks):
    packer.pack_ext_type(1, chunk)  # caller tracks ordering

MSGPACK_MAX_EXT_DATA = 0xFFFFFFFF

def is_serializable_ext_data(data: bytes) -> bool:
    return len(data) <= MSGPACK_MAX_EXT_DATA

def is_small_enough_ext_data(data) -> bool:
    return isinstance(data, bytes) and len(data) <= 0xFFFFFFFF

try:
    packer.pack_ext_type(code, data)
except ValueError as e:
    if 'Too large' in str(e):
        # split and pack as multiple ext messages
        for i in range(0, len(data), 0xFFFFFFFF):
            packer.pack_ext_type(code, data[i:i+0xFFFFFFFF])
    else:
        raise

When it happens

Trigger: Calling packer.pack_ext_type(code, data) where len(data) > 0xFFFFFFFF (4,294,967,295 bytes). Encountered when embedding large blobs as custom extension types.

Common situations: Serializing large binary blobs (media files, serialized models) as a single ext type; insufficient chunking strategy for extension data.

Related errors

• Memoryview is too large
• typecode must have int type.
• typecode should be 0-127
• data must have bytes type
• Array is too large

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/44de6dc24739cf28. Report an issue: GitHub.