{"record":{"id":"24589738ebc8c442","repo":"pypa/pip","slug":"bin-is-too-large","errorCode":null,"errorMessage":"Bin is too large","messagePattern":"Bin is too large","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/msgpack/fallback.py","lineNumber":921,"sourceCode":"            self._buffer.write(struct.pack(\">BB\", 0xD9, n))\n        elif n <= 0xFFFF:\n            self._buffer.write(struct.pack(\">BH\", 0xDA, n))\n        elif n <= 0xFFFFFFFF:\n            self._buffer.write(struct.pack(\">BI\", 0xDB, n))\n        else:\n            raise ValueError(\"Raw is too large\")\n\n    def _pack_bin_header(self, n):\n        if not self._use_bin_type:\n            return self._pack_raw_header(n)\n        elif n <= 0xFF:\n            return self._buffer.write(struct.pack(\">BB\", 0xC4, n))\n        elif n <= 0xFFFF:\n            return self._buffer.write(struct.pack(\">BH\", 0xC5, n))\n        elif n <= 0xFFFFFFFF:\n            return self._buffer.write(struct.pack(\">BI\", 0xC6, n))\n        else:\n            raise ValueError(\"Bin is too large\")\n\n    def bytes(self):\n        \"\"\"Return internal buffer contents as bytes object\"\"\"\n        return self._buffer.getvalue()\n\n    def reset(self):\n        \"\"\"Reset internal buffer.\n\n        This method is useful only when autoreset=False.\n        \"\"\"\n        self._buffer = BytesIO()\n\n    def getbuffer(self):\n        \"\"\"Return view of internal buffer.\"\"\"\n        if _USING_STRINGBUILDER:\n            return memoryview(self.bytes())\n        else:\n            return self._buffer.getbuffer()","sourceCodeStart":903,"sourceCodeEnd":939,"githubUrl":"https://github.com/pypa/pip/blob/f399c3718970b1b0e2478dac5296eb62679a9b86/src/pip/_vendor/msgpack/fallback.py#L903-L939","documentation":"msgpack's pure-Python Packer (fallback.py) encodes binary data using the 'bin' type family. _pack_bin_header selects a header format by length: 8-bit (<=255), 16-bit (<=65535), or 32-bit (<=0xFFFFFFFF). The msgpack spec has no 64-bit bin type, so any bytes object exceeding 4 GiB cannot be encoded and raises ValueError. This occurs only when _use_bin_type=True (the modern default).","triggerScenarios":"Calling msgpack.packb(data) or Packer().pack(data) where a value (or nested value inside a list/dict) is a bytes/bytearray object larger than 4,294,967,295 (0xFFFFFFFF) bytes.","commonSituations":"Serializing large media files, database dumps, ML model weights, or accumulated memory-mapped buffers whole into a single msgpack payload instead of chunking.","solutions":["Split the large bytes into chunks smaller than 4 GiB and pack as a list of chunks with a total-size field","Stream the data to disk or network in frames rather than serializing the entire payload at once","Apply compression (zlib/zstd) before packing to bring the payload under the limit","Switch to a serialization format that supports 64-bit length fields (e.g., CBOR)"],"exampleFix":"// before\ndata = open(\"huge_model.bin\", \"rb\read()  # 6 GiB\npacked = msgpack.packb({\"model\": data})  # raises \"Bin is too large\"\n\n// after\nCHUNK = 256 * 1024 * 1024  # 256 MiB\nchunks = [data[i:i+CHUNK] for i in range(0, len(data), CHUNK)]\npacked = msgpack.packb({\"model_chunks\": chunks, \"total_size\": len(data)})","handlingStrategy":"validation","validationCode":"MAX_BIN = 0xFFFFFFFF  # 4 GiB msgpack bin type limit\n\ndef check_bin_sizes(obj, path=\"root\"):\n    if isinstance(obj, (bytes, bytearray)):\n        if len(obj) > MAX_BIN:\n            raise ValueError(f\"Bytes at {path} is {len(obj)} bytes, exceeds msgpack max {MAX_BIN}\")\n    elif isinstance(obj, dict):\n        for k, v in obj.items():\n            check_bin_sizes(v, f\"{path}.{k}\")\n    elif isinstance(obj, (list, tuple)):\n        for i, v in enumerate(obj):\n            check_bin_sizes(v, f\"{path}[{i}]\")","typeGuard":null,"tryCatchPattern":"try:\n    packed = msgpack.packb(data)\nexcept ValueError as e:\n    if \"Bin is too large\" in str(e):\n        # chunk or compress the oversized bytes value\n        raise","preventionTips":["Always chunk large binary payloads before serialization","Monitor payload sizes in logging before packing","Consider compression for large data","Use streaming packers for data exceeding 1 GiB"],"tags":["msgpack","serialization","data-size","vendored"],"backgroundTag":null,"analyzedSha":"f399c3718970b1b0e2478dac5296eb62679a9b86","analyzedAt":"2026-08-08T23:01:42.227Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}