pypa/pip · error · ValueError

Raw is too large

Error message

Raw is too large

What it means

Raised by msgpack's pure-Python Packer._pack_raw_header when the raw string length exceeds 0xFFFFFFFF (4,294,967,295 bytes). The raw/str format encodes length in a 32-bit field, so strings whose UTF-8 encoding exceeds ~4 GiB cannot be represented. This is the string/raw-type-specific size limit, distinct from the bin type limit.

Source

Thrown at src/pip/_vendor/msgpack/fallback.py:909

        raise ValueError("Dict is too large")

    def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT):
        self._pack_map_header(n)
        for k, v in pairs:
            self._pack(k, nest_limit - 1)
            self._pack(v, nest_limit - 1)

    def _pack_raw_header(self, n):
        if n <= 0x1F:
            self._buffer.write(struct.pack("B", 0xA0 + n))
        elif self._use_bin_type and n <= 0xFF:
            self._buffer.write(struct.pack(">BB", 0xD9, n))
        elif n <= 0xFFFF:
            self._buffer.write(struct.pack(">BH", 0xDA, n))
        elif n <= 0xFFFFFFFF:
            self._buffer.write(struct.pack(">BI", 0xDB, n))
        else:
            raise ValueError("Raw is too large")

    def _pack_bin_header(self, n):
        if not self._use_bin_type:
            return self._pack_raw_header(n)
        elif n <= 0xFF:
            return self._buffer.write(struct.pack(">BB", 0xC4, n))
        elif n <= 0xFFFF:
            return self._buffer.write(struct.pack(">BH", 0xC5, n))
        elif n <= 0xFFFFFFFF:
            return self._buffer.write(struct.pack(">BI", 0xC6, n))
        else:
            raise ValueError("Bin is too large")

    def bytes(self):
        """Return internal buffer contents as bytes object"""
        return self._buffer.getvalue()

    def reset(self):

View on GitHub (pinned to f399c37189)

Solutions

1. Encode the string to bytes and chunk into multiple bin messages each under 4 GiB.
2. Store large text externally and pack only a reference.
3. Pre-check len(s.encode('utf-8')) before packing.
4. Use a streaming approach for very large text data.

Example fix

# before
packb(huge_string)  # raises if UTF-8 encoding > 4 GiB

# after — encode and chunk
encoded = huge_string.encode('utf-8')
MAX = 0xFFFFFFFF
chunks = [encoded[i:i+MAX] for i in range(0, len(encoded), MAX)]
packb([c for c in chunks])  # reassemble on unpack

MSGPACK_MAX_STR = 0xFFFFFFFF

def is_serializable_string(s: str) -> bool:
    return len(s.encode('utf-8')) < MSGPACK_MAX_STR

def is_serializable_str(obj) -> bool:
    return isinstance(obj, str) and len(obj.encode('utf-8')) < 0xFFFFFFFF

try:
    packed = packer.pack(huge_string)
except ValueError as e:
    if 'too large' in str(e).lower():
        encoded = huge_string.encode('utf-8')
        chunks = [encoded[i:i+0xFFFFFFFF] for i in range(0, len(encoded), 0xFFFFFFFF)]
        packed = packer.pack([c for c in chunks])
    else:
        raise

When it happens

Trigger: Packing a str whose UTF-8 encoded byte length exceeds 0xFFFFFFFF. Hit via packer.pack(very_long_string) or packb(very_long_string). Also reached internally when _pack_bin_header falls through to _pack_raw_header in non-bin-type mode.

Common situations: Serializing very large text payloads (logs, JSON, base64 data, generated source code) as a single msgpack string; embedding large file contents as strings.

Related errors

• Memoryview is too large
• Array is too large
• Dict is too large
• Cannot serialize {obj!r} where tzinfo=None
• Cannot serialize {obj!r}

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/add54e59745111ce. Report an issue: GitHub.