pypa/pip · error · ValueError

Memoryview is too large

Error message

Memoryview is too large

What it means

Raised by msgpack's pure-Python Packer._pack when serializing a memoryview whose total byte length (obj.nbytes) is >= 2^32 (4 GiB). The msgpack binary format uses a 32-bit length field for bin types, so payloads at or above 4 GiB cannot be represented.

Source

Thrown at src/pip/_vendor/msgpack/fallback.py:746

                    continue
                raise OverflowError("Integer value out of range")
            if check(obj, (bytes, bytearray)):
                n = len(obj)
                if n >= 2**32:
                    raise ValueError("%s is too large" % type(obj).__name__)
                self._pack_bin_header(n)
                return self._buffer.write(obj)
            if check(obj, str):
                obj = obj.encode("utf-8", self._unicode_errors)
                n = len(obj)
                if n >= 2**32:
                    raise ValueError("String is too large")
                self._pack_raw_header(n)
                return self._buffer.write(obj)
            if check(obj, memoryview):
                n = obj.nbytes
                if n >= 2**32:
                    raise ValueError("Memoryview is too large")
                self._pack_bin_header(n)
                return self._buffer.write(obj)
            if check(obj, float):
                if self._use_float:
                    return self._buffer.write(struct.pack(">Bf", 0xCA, obj))
                return self._buffer.write(struct.pack(">Bd", 0xCB, obj))
            if check(obj, (ExtType, Timestamp)):
                if check(obj, Timestamp):
                    code = -1
                    data = obj.to_bytes()
                else:
                    code = obj.code
                    data = obj.data
                assert isinstance(code, int)
                assert isinstance(data, bytes)
                L = len(data)
                if L == 1:
                    self._buffer.write(b"\xd4")

View on GitHub (pinned to f399c37189)

Solutions

1. Chunk the data into pieces smaller than 4 GiB and pack as an array of smaller bin objects.
2. Use a different serialization format that supports 64-bit lengths if you genuinely need >4 GiB blobs.
3. Store the large blob externally (e.g. file/S3) and pack only a reference.
4. Convert to multiple smaller memoryviews before packing.

Example fix

# before
packer = msgpack.Packer()
packer.pack(large_memview)  # raises if >= 4 GiB

# after — chunk into <=1 GiB pieces
chunk_size = 1024**3
chunks = [large_memview[i:i+chunk_size] for i in range(0, len(large_memview), chunk_size)]
packer.pack([bytes(c) for c in chunks])

MSGPACK_MAX_BIN = 2**32

def is_serializable_memview(mv: memoryview) -> bool:
    return mv.nbytes < MSGPACK_MAX_BIN

def is_small_enough_memview(mv) -> bool:
    return isinstance(mv, memoryview) and mv.nbytes < 2**32

try:
    packer.pack(large_mv)
except ValueError as e:
    if 'too large' in str(e).lower():
        # chunk and pack as array
        chunks = [large_mv[i:i+CHUNK] for i in range(0, len(large_mv), CHUNK)]
        packer.pack([bytes(c) for c in chunks])
    else:
        raise

When it happens

Trigger: Packing a memoryview, numpy array buffer, mmap, or large byte slice whose underlying data is 4 GiB or larger via msgpack.Packer().pack(obj) or packb(memoryview_obj).

Common situations: Serializing large scientific/array data (numpy, torch tensors) to msgpack; embedding large file contents; insufficient chunking of big payloads.

Related errors

• Array is too large
• Dict is too large
• Raw is too large
• Cannot serialize {obj!r} where tzinfo=None
• Cannot serialize {obj!r}

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/610817997c70f857. Report an issue: GitHub.