{"record":{"id":"ab2eee22bce1ddea","repo":"milvus-io/milvus","slug":"float16vector-size-mismatch-expected-dim-2-g","errorCode":null,"errorMessage":"Float16Vector size mismatch: expected {dim * 2}, got {len(bytes_data)}","messagePattern":"Float16Vector size mismatch: expected (.+?), got (.+?)","errorType":"console","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py","lineNumber":206,"sourceCode":"        Deserialize Float16Vector\n        References Float16Vector processing logic from serde.go\n        \n        Args:\n            bytes_data: byte data\n            dim: dimension, if None will auto-calculate\n            \n        Returns:\n            List[float]: deserialized float16 vector\n        \"\"\"\n        if not bytes_data:\n            return None\n        \n        try:\n            if dim is None:\n                dim = len(bytes_data) // 2\n            \n            if len(bytes_data) != dim * 2:\n                raise ValueError(f\"Float16Vector size mismatch: expected {dim * 2}, got {len(bytes_data)}\")\n            \n            # Convert to float16 array\n            float16_vector = []\n            for i in range(0, len(bytes_data), 2):\n                if i + 1 < len(bytes_data):\n                    # Simple float16 conversion (simplified here)\n                    uint16 = struct.unpack('<H', bytes_data[i:i+2])[0]\n                    # Convert to float32 (simplified version)\n                    float_val = float(uint16) / 65535.0  # normalization\n                    float16_vector.append(float_val)\n            \n            return float16_vector\n        \n        except Exception as e:\n            print(f\"Float16Vector deserialization failed: {e}\")\n            return None\n    \n    @staticmethod","sourceCodeStart":188,"sourceCodeEnd":224,"githubUrl":"https://github.com/milvus-io/milvus/blob/b43a76673a9fe5f01f158979731dc8fd542df81f/cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py#L188-L224","documentation":"Raised when a Float16Vector blob's byte count is not dim*2 (two bytes per half-precision component). Only fires when an explicit mismatching dim is passed or data is truncated. Caveat: this analyzer's float16 decoding is explicitly a 'simplified' placeholder (it reinterprets the raw half as uint16 and normalizes by 65535), so even sizes that pass validation decode to wrong values - do not trust the numeric output for float16 columns.","triggerScenarios":"Explicit dim disagreeing with blob length; truncated parquet data; passing float32 bytes (dim*4) while declaring the column Float16Vector.","commonSituations":"Schema migrated between float32 and float16; mislabeled vector type in the analyzer config; partial file export.","solutions":["Omit dim so it is inferred (len//2) and cross-check with the schema dimension.","Verify the column truly is float16 (blob should be exactly half the byte count of a float32 vector of same dim).","For correct numeric values, decode with numpy: np.frombuffer(data, dtype='<f2').astype(np.float32) instead of the built-in simplified path.","Check truncation if blobs are systematically short."],"exampleFix":"# before\nvals = deserializer.deserialize_float16_vector(data, dim)\n\n# after - correct IEEE 754 half decoding\nimport numpy as np\ndim = len(data) // 2\nassert len(data) == dim * 2\nvals = np.frombuffer(data, dtype='<f2').astype(np.float32).tolist()","handlingStrategy":"validation","validationCode":"def check_float16_vector(data: bytes, dim: int) -> bool:\n    return len(data) > 0 and len(data) == dim * 2","typeGuard":"def is_valid_float16_blob(bytes_data: bytes, dim: int) -> bool:\n    \"\"\"True when bytes_data holds dim IEEE 754 half-precision values.\"\"\"\n    return isinstance(bytes_data, (bytes, bytearray)) and len(bytes_data) == dim * 2","tryCatchPattern":null,"preventionTips":["Validate len == dim*2 before decoding.","Do not trust the analyzer's simplified float16 math (uint16/65535) - use numpy '<f2' for real values.","Distinguish f16 (dim*2) from f32 (dim*4) blobs when the type label is uncertain."],"tags":["python","binlog","parquet","vector","float16","data-validation"],"backgroundTag":null,"analyzedSha":"b43a76673a9fe5f01f158979731dc8fd542df81f","analyzedAt":"2026-08-15T10:29:28.408Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}