{"record":{"id":"610817997c70f857","repo":"pypa/pip","slug":"memoryview-is-too-large","errorCode":null,"errorMessage":"Memoryview is too large","messagePattern":"Memoryview is too large","errorType":"exception","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"src/pip/_vendor/msgpack/fallback.py","lineNumber":746,"sourceCode":"                    continue\n                raise OverflowError(\"Integer value out of range\")\n            if check(obj, (bytes, bytearray)):\n                n = len(obj)\n                if n >= 2**32:\n                    raise ValueError(\"%s is too large\" % type(obj).__name__)\n                self._pack_bin_header(n)\n                return self._buffer.write(obj)\n            if check(obj, str):\n                obj = obj.encode(\"utf-8\", self._unicode_errors)\n                n = len(obj)\n                if n >= 2**32:\n                    raise ValueError(\"String is too large\")\n                self._pack_raw_header(n)\n                return self._buffer.write(obj)\n            if check(obj, memoryview):\n                n = obj.nbytes\n                if n >= 2**32:\n                    raise ValueError(\"Memoryview is too large\")\n                self._pack_bin_header(n)\n                return self._buffer.write(obj)\n            if check(obj, float):\n                if self._use_float:\n                    return self._buffer.write(struct.pack(\">Bf\", 0xCA, obj))\n                return self._buffer.write(struct.pack(\">Bd\", 0xCB, obj))\n            if check(obj, (ExtType, Timestamp)):\n                if check(obj, Timestamp):\n                    code = -1\n                    data = obj.to_bytes()\n                else:\n                    code = obj.code\n                    data = obj.data\n                assert isinstance(code, int)\n                assert isinstance(data, bytes)\n                L = len(data)\n                if L == 1:\n                    self._buffer.write(b\"\\xd4\")","sourceCodeStart":728,"sourceCodeEnd":764,"githubUrl":"https://github.com/pypa/pip/blob/f399c3718970b1b0e2478dac5296eb62679a9b86/src/pip/_vendor/msgpack/fallback.py#L728-L764","documentation":"Raised by msgpack's pure-Python Packer._pack when serializing a memoryview whose total byte length (obj.nbytes) is >= 2^32 (4 GiB). The msgpack binary format uses a 32-bit length field for bin types, so payloads at or above 4 GiB cannot be represented.","triggerScenarios":"Packing a memoryview, numpy array buffer, mmap, or large byte slice whose underlying data is 4 GiB or larger via msgpack.Packer().pack(obj) or packb(memoryview_obj).","commonSituations":"Serializing large scientific/array data (numpy, torch tensors) to msgpack; embedding large file contents; insufficient chunking of big payloads.","solutions":["Chunk the data into pieces smaller than 4 GiB and pack as an array of smaller bin objects.","Use a different serialization format that supports 64-bit lengths if you genuinely need >4 GiB blobs.","Store the large blob externally (e.g. file/S3) and pack only a reference.","Convert to multiple smaller memoryviews before packing."],"exampleFix":"# before\npacker = msgpack.Packer()\npacker.pack(large_memview)  # raises if >= 4 GiB\n\n# after — chunk into <=1 GiB pieces\nchunk_size = 1024**3\nchunks = [large_memview[i:i+chunk_size] for i in range(0, len(large_memview), chunk_size)]\npacker.pack([bytes(c) for c in chunks])","handlingStrategy":"validation","validationCode":"MSGPACK_MAX_BIN = 2**32\n\ndef is_serializable_memview(mv: memoryview) -> bool:\n    return mv.nbytes < MSGPACK_MAX_BIN","typeGuard":"def is_small_enough_memview(mv) -> bool:\n    return isinstance(mv, memoryview) and mv.nbytes < 2**32","tryCatchPattern":"try:\n    packer.pack(large_mv)\nexcept ValueError as e:\n    if 'too large' in str(e).lower():\n        # chunk and pack as array\n        chunks = [large_mv[i:i+CHUNK] for i in range(0, len(large_mv), CHUNK)]\n        packer.pack([bytes(c) for c in chunks])\n    else:\n        raise","preventionTips":["Check memoryview.nbytes < 4 GiB before packing.","Chunk large binary payloads before serialization.","Store very large blobs externally and serialize references."],"tags":["msgpack","serialization","size-limit","memoryview"],"backgroundTag":null,"analyzedSha":"f399c3718970b1b0e2478dac5296eb62679a9b86","analyzedAt":"2026-08-08T23:01:42.227Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}