pypa/pip · error · ValueError

Bin is too large

Error message

Bin is too large

What it means

msgpack's pure-Python Packer (fallback.py) encodes binary data using the 'bin' type family. _pack_bin_header selects a header format by length: 8-bit (<=255), 16-bit (<=65535), or 32-bit (<=0xFFFFFFFF). The msgpack spec has no 64-bit bin type, so any bytes object exceeding 4 GiB cannot be encoded and raises ValueError. This occurs only when _use_bin_type=True (the modern default).

Source

Thrown at src/pip/_vendor/msgpack/fallback.py:921

            self._buffer.write(struct.pack(">BB", 0xD9, n))
        elif n <= 0xFFFF:
            self._buffer.write(struct.pack(">BH", 0xDA, n))
        elif n <= 0xFFFFFFFF:
            self._buffer.write(struct.pack(">BI", 0xDB, n))
        else:
            raise ValueError("Raw is too large")

    def _pack_bin_header(self, n):
        if not self._use_bin_type:
            return self._pack_raw_header(n)
        elif n <= 0xFF:
            return self._buffer.write(struct.pack(">BB", 0xC4, n))
        elif n <= 0xFFFF:
            return self._buffer.write(struct.pack(">BH", 0xC5, n))
        elif n <= 0xFFFFFFFF:
            return self._buffer.write(struct.pack(">BI", 0xC6, n))
        else:
            raise ValueError("Bin is too large")

    def bytes(self):
        """Return internal buffer contents as bytes object"""
        return self._buffer.getvalue()

    def reset(self):
        """Reset internal buffer.

        This method is useful only when autoreset=False.
        """
        self._buffer = BytesIO()

    def getbuffer(self):
        """Return view of internal buffer."""
        if _USING_STRINGBUILDER:
            return memoryview(self.bytes())
        else:
            return self._buffer.getbuffer()

View on GitHub (pinned to f399c37189)

Solutions

1. Split the large bytes into chunks smaller than 4 GiB and pack as a list of chunks with a total-size field
2. Stream the data to disk or network in frames rather than serializing the entire payload at once
3. Apply compression (zlib/zstd) before packing to bring the payload under the limit
4. Switch to a serialization format that supports 64-bit length fields (e.g., CBOR)

Example fix

// before
data = open("huge_model.bin", "rb
ead()  # 6 GiB
packed = msgpack.packb({"model": data})  # raises "Bin is too large"

// after
CHUNK = 256 * 1024 * 1024  # 256 MiB
chunks = [data[i:i+CHUNK] for i in range(0, len(data), CHUNK)]
packed = msgpack.packb({"model_chunks": chunks, "total_size": len(data)})

MAX_BIN = 0xFFFFFFFF  # 4 GiB msgpack bin type limit

def check_bin_sizes(obj, path="root"):
    if isinstance(obj, (bytes, bytearray)):
        if len(obj) > MAX_BIN:
            raise ValueError(f"Bytes at {path} is {len(obj)} bytes, exceeds msgpack max {MAX_BIN}")
    elif isinstance(obj, dict):
        for k, v in obj.items():
            check_bin_sizes(v, f"{path}.{k}")
    elif isinstance(obj, (list, tuple)):
        for i, v in enumerate(obj):
            check_bin_sizes(v, f"{path}[{i}]")

try:
    packed = msgpack.packb(data)
except ValueError as e:
    if "Bin is too large" in str(e):
        # chunk or compress the oversized bytes value
        raise

When it happens

Trigger: Calling msgpack.packb(data) or Packer().pack(data) where a value (or nested value inside a list/dict) is a bytes/bytearray object larger than 4,294,967,295 (0xFFFFFFFF) bytes.

Common situations: Serializing large media files, database dumps, ML model weights, or accumulated memory-mapped buffers whole into a single msgpack payload instead of chunking.

Related errors

• Memoryview is too large
• Cannot serialize {obj!r} where tzinfo=None
• Cannot serialize {obj!r}
• Array is too large
• Dict is too large

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/24589738ebc8c442. Report an issue: GitHub.