pypa/pip · error · ValueError

Dict is too large

Error message

Dict is too large

What it means

Raised by msgpack's pure-Python Packer._pack_map_header when the dictionary key-value pair count exceeds 0xFFFFFFFF (4,294,967,295). The msgpack map header uses at most a 32-bit length field, so maps with more than ~4.29 billion entries cannot be encoded.

Source

Thrown at src/pip/_vendor/msgpack/fallback.py:891

            return ret

    def _pack_array_header(self, n):
        if n <= 0x0F:
            return self._buffer.write(struct.pack("B", 0x90 + n))
        if n <= 0xFFFF:
            return self._buffer.write(struct.pack(">BH", 0xDC, n))
        if n <= 0xFFFFFFFF:
            return self._buffer.write(struct.pack(">BI", 0xDD, n))
        raise ValueError("Array is too large")

    def _pack_map_header(self, n):
        if n <= 0x0F:
            return self._buffer.write(struct.pack("B", 0x80 + n))
        if n <= 0xFFFF:
            return self._buffer.write(struct.pack(">BH", 0xDE, n))
        if n <= 0xFFFFFFFF:
            return self._buffer.write(struct.pack(">BI", 0xDF, n))
        raise ValueError("Dict is too large")

    def _pack_map_pairs(self, n, pairs, nest_limit=DEFAULT_RECURSE_LIMIT):
        self._pack_map_header(n)
        for k, v in pairs:
            self._pack(k, nest_limit - 1)
            self._pack(v, nest_limit - 1)

    def _pack_raw_header(self, n):
        if n <= 0x1F:
            self._buffer.write(struct.pack("B", 0xA0 + n))
        elif self._use_bin_type and n <= 0xFF:
            self._buffer.write(struct.pack(">BB", 0xD9, n))
        elif n <= 0xFFFF:
            self._buffer.write(struct.pack(">BH", 0xDA, n))
        elif n <= 0xFFFFFFFF:
            self._buffer.write(struct.pack(">BI", 0xDB, n))
        else:
            raise ValueError("Raw is too large")

View on GitHub (pinned to f399c37189)

Solutions

1. Split the map into multiple messages each under the limit.
2. Stream key-value pairs using msgpack's streaming Packer rather than building one map.
3. Use a database or external key-value store for very large mappings and serialize only a reference.
4. Pre-check len(d) and reject/batch if it exceeds the limit.

Example fix

# before
packb(huge_dict)  # raises if len > 0xFFFFFFFF

# after — batch into smaller maps
batch_size = 1_000_000
items = list(huge_dict.items())
for i in range(0, len(items), batch_size):
    batch = dict(items[i:i+batch_size])
    stream.write(msgpack.packb(batch))

MSGPACK_MAX_MAP = 0xFFFFFFFF

def is_serializable_dict(d) -> bool:
    return len(d) < MSGPACK_MAX_MAP

def is_serializable_dict(obj) -> bool:
    return isinstance(obj, dict) and len(obj) < 0xFFFFFFFF

try:
    packed = packer.pack(huge_dict)
except ValueError as e:
    if 'too large' in str(e).lower():
        items = list(huge_dict.items())
        for i in range(0, len(items), 1_000_000):
            stream.write(packer.pack(dict(items[i:i+1_000_000])))
    else:
        raise

When it happens

Trigger: Packing a dict with more than 0xFFFFFFFF entries via packer.pack(obj) or packb(obj). Also raised via pack_map_pairs or pack_map_header when the count is oversized.

Common situations: Serializing a massive key-value store, cache dump, or index as a single msgpack map; unbounded accumulation of entries; in-memory aggregation without eviction.

Related errors

• Memoryview is too large
• Array is too large
• Raw is too large
• Cannot serialize {obj!r} where tzinfo=None
• Cannot serialize {obj!r}

AI-assisted analysis of pypa/pip@f399c37189 (2026-08-08). Data as JSON: /api/errors/1cd08be5241a5824. Report an issue: GitHub.