{"record":{"id":"726d3fae8c215046","repo":"milvus-io/milvus","slug":"binaryvector-size-mismatch-expected-expected-siz","errorCode":null,"errorMessage":"BinaryVector size mismatch: expected {expected_size}, got {len(bytes_data)}","messagePattern":"BinaryVector size mismatch: expected (.+?), got (.+?)","errorType":"console","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py","lineNumber":138,"sourceCode":"        References BinaryVector processing logic from serde.go\n        \n        Args:\n            bytes_data: byte data\n            dim: dimension, if None will auto-calculate\n            \n        Returns:\n            List[int]: deserialized binary vector\n        \"\"\"\n        if not bytes_data:\n            return None\n        \n        try:\n            if dim is None:\n                dim = len(bytes_data) * 8\n            \n            expected_size = (dim + 7) // 8\n            if len(bytes_data) != expected_size:\n                raise ValueError(f\"BinaryVector size mismatch: expected {expected_size}, got {len(bytes_data)}\")\n            \n            # Convert to binary vector\n            binary_vector = []\n            for byte in bytes_data:\n                for i in range(8):\n                    bit = (byte >> i) & 1\n                    binary_vector.append(bit)\n            \n            # Only return first dim elements\n            return binary_vector[:dim]\n        \n        except Exception as e:\n            print(f\"BinaryVector deserialization failed: {e}\")\n            return None\n    \n    @staticmethod\n    def deserialize_int8_vector(bytes_data: bytes, dim: Optional[int] = None) -> Optional[List[int]]:\n        \"\"\"","sourceCodeStart":120,"sourceCodeEnd":156,"githubUrl":"https://github.com/milvus-io/milvus/blob/b43a76673a9fe5f01f158979731dc8fd542df81f/cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py#L120-L156","documentation":"Raised when a BinaryVector blob's byte length does not equal ceil(dim/8). Binary vectors pack dim bits, 8 per byte, MSB-first per Milvus storage; the analyzer validates the byte count against the schema dimension and rejects mismatches. Note the deserializer itself infers dim = len*8 when dim is None, so this fires only when an explicit wrong dim is supplied or the blob is truncated.","triggerScenarios":"Passing an explicit dim that disagrees with the blob length (e.g. dim=100 for 13 bytes, since ceil(100/8)=13 matches only specific values); truncated parquet data; column mislabeled as BinaryVector.","commonSituations":"Schema dim changed after files were written; manual dim entry on the command line; reading mixed-type vector columns with one blanket type.","solutions":["Omit the explicit dim argument and let it be inferred from the byte length, then cross-check against the schema.","Verify dim from collection schema and confirm ceil(dim/8) equals the blob length.","Check the parquet file for truncation if lengths are systematically short."],"exampleFix":"# before\ndeserializer.deserialize_binary_vector(data, dim=64)  # blob is 16 bytes\n\n# after - let dim be inferred, or validate first\ndim = len(data) * 8\nexpected = (dim + 7) // 8\nassert len(data) == expected\ndeserializer.deserialize_binary_vector(data, dim)","handlingStrategy":"validation","validationCode":"import math\ndef check_binary_vector(data: bytes, dim: int) -> bool:\n    return len(data) > 0 and len(data) == math.ceil(dim / 8)","typeGuard":"def is_valid_binary_vector_blob(bytes_data: bytes, dim: int) -> bool:\n    \"\"\"True when bytes_data holds exactly dim bits, 8 per byte.\"\"\"\n    import math\n    return isinstance(bytes_data, (bytes, bytearray)) and len(bytes_data) == math.ceil(dim / 8)","tryCatchPattern":null,"preventionTips":["Prefer omitting dim so it is inferred from blob length, then compare against the schema.","Remember binary vectors are bit-packed: expected bytes = ceil(dim/8), not dim.","Verify parquet exports completed fully; truncation is the top cause of size mismatches."],"tags":["python","binlog","parquet","vector","data-validation"],"backgroundTag":null,"analyzedSha":"b43a76673a9fe5f01f158979731dc8fd542df81f","analyzedAt":"2026-08-15T10:29:28.408Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}