{"record":{"id":"1caef2b626b7d052","repo":"milvus-io/milvus","slug":"bfloat16vector-size-mismatch-expected-dim-2","errorCode":null,"errorMessage":"BFloat16Vector size mismatch: expected {dim * 2}, got {len(bytes_data)}","messagePattern":"BFloat16Vector size mismatch: expected (.+?), got (.+?)","errorType":"console","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py","lineNumber":245,"sourceCode":"        Deserialize BFloat16Vector\n        References BFloat16Vector processing logic from serde.go\n        \n        Args:\n            bytes_data: byte data\n            dim: dimension, if None will auto-calculate\n            \n        Returns:\n            List[float]: deserialized bfloat16 vector\n        \"\"\"\n        if not bytes_data:\n            return None\n        \n        try:\n            if dim is None:\n                dim = len(bytes_data) // 2\n            \n            if len(bytes_data) != dim * 2:\n                raise ValueError(f\"BFloat16Vector size mismatch: expected {dim * 2}, got {len(bytes_data)}\")\n            \n            # Convert to bfloat16 array\n            bfloat16_vector = []\n            for i in range(0, len(bytes_data), 2):\n                if i + 1 < len(bytes_data):\n                    # Simple bfloat16 conversion (simplified here)\n                    uint16 = struct.unpack('<H', bytes_data[i:i+2])[0]\n                    # Convert to float32 (simplified version)\n                    float_val = float(uint16) / 65535.0  # normalization\n                    bfloat16_vector.append(float_val)\n            \n            return bfloat16_vector\n        \n        except Exception as e:\n            print(f\"BFloat16Vector deserialization failed: {e}\")\n            return None\n    \n    @staticmethod","sourceCodeStart":227,"sourceCodeEnd":263,"githubUrl":"https://github.com/milvus-io/milvus/blob/b43a76673a9fe5f01f158979731dc8fd542df81f/cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py#L227-L263","documentation":"Raised when a BFloat16Vector blob's byte count is not dim*2. Same shape as the float16 case: explicit wrong dim or truncated data triggers it. The built-in decode is likewise a placeholder (uint16/65535 normalization), which is not bfloat16 semantics - bfloat16 is a truncated float32 (upper 16 bits), so passing validation still yields wrong numbers.","triggerScenarios":"Explicit dim mismatch; truncated blob; feeding float32 or float16 bytes while declaring BFloat16Vector.","commonSituations":"Index/vector type changed after files were written; mislabeled column; incomplete export.","solutions":["Omit dim so it is inferred as len//2 and verify against schema.","Decode correctly for analysis: shift the 16-bit word left by 16 into a uint32 view and reinterpret as float32.","Verify blob length relationships to distinguish bf16 (dim*2) from f32 (dim*4) data if the type label is suspect.","Re-export if data is truncated."],"exampleFix":"# before\nvals = deserializer.deserialize_bfloat16_vector(data, dim)\n\n# after - true bfloat16 decode\nimport numpy as np\ndim = len(data) // 2\nassert len(data) == dim * 2\nu16 = np.frombuffer(data, dtype='<u2').astype(np.uint32)\nvals = (u16 << 16).view(np.float32).tolist()","handlingStrategy":"validation","validationCode":"def check_bfloat16_vector(data: bytes, dim: int) -> bool:\n    return len(data) > 0 and len(data) == dim * 2","typeGuard":"def is_valid_bfloat16_blob(bytes_data: bytes, dim: int) -> bool:\n    \"\"\"True when bytes_data holds dim bfloat16 (truncated float32) values.\"\"\"\n    return isinstance(bytes_data, (bytes, bytearray)) and len(bytes_data) == dim * 2","tryCatchPattern":null,"preventionTips":["Validate len == dim*2 before decoding.","For correct values decode bfloat16 by bit-shifting: (u16 << 16).view(float32); the built-in normalization is a placeholder.","Confirm the column type label matches the actual encoding (bf16 vs f16 vs f32) before analysis."],"tags":["python","binlog","parquet","vector","bfloat16","data-validation"],"backgroundTag":null,"analyzedSha":"b43a76673a9fe5f01f158979731dc8fd542df81f","analyzedAt":"2026-08-15T10:29:28.408Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}