milvus-io/milvus · error · ValueError

BFloat16Vector size mismatch: expected {dim * 2}, got {len(b

Error message

BFloat16Vector size mismatch: expected {dim * 2}, got {len(bytes_data)}

What it means

Raised when a BFloat16Vector blob's byte count is not dim*2. Same shape as the float16 case: explicit wrong dim or truncated data triggers it. The built-in decode is likewise a placeholder (uint16/65535 normalization), which is not bfloat16 semantics - bfloat16 is a truncated float32 (upper 16 bits), so passing validation still yields wrong numbers.

Source

Thrown at cmd/tools/binlogv2/parquet_analyzer/vector_deserializer.py:245

        Deserialize BFloat16Vector
        References BFloat16Vector processing logic from serde.go
        
        Args:
            bytes_data: byte data
            dim: dimension, if None will auto-calculate
            
        Returns:
            List[float]: deserialized bfloat16 vector
        """
        if not bytes_data:
            return None
        
        try:
            if dim is None:
                dim = len(bytes_data) // 2
            
            if len(bytes_data) != dim * 2:
                raise ValueError(f"BFloat16Vector size mismatch: expected {dim * 2}, got {len(bytes_data)}")
            
            # Convert to bfloat16 array
            bfloat16_vector = []
            for i in range(0, len(bytes_data), 2):
                if i + 1 < len(bytes_data):
                    # Simple bfloat16 conversion (simplified here)
                    uint16 = struct.unpack('<H', bytes_data[i:i+2])[0]
                    # Convert to float32 (simplified version)
                    float_val = float(uint16) / 65535.0  # normalization
                    bfloat16_vector.append(float_val)
            
            return bfloat16_vector
        
        except Exception as e:
            print(f"BFloat16Vector deserialization failed: {e}")
            return None
    
    @staticmethod

View on GitHub (pinned to b43a76673a)

Solutions

Omit dim so it is inferred as len//2 and verify against schema.
Decode correctly for analysis: shift the 16-bit word left by 16 into a uint32 view and reinterpret as float32.
Verify blob length relationships to distinguish bf16 (dim*2) from f32 (dim*4) data if the type label is suspect.
Re-export if data is truncated.

Example fix

# before
vals = deserializer.deserialize_bfloat16_vector(data, dim)

# after - true bfloat16 decode
import numpy as np
dim = len(data) // 2
assert len(data) == dim * 2
u16 = np.frombuffer(data, dtype='<u2').astype(np.uint32)
vals = (u16 << 16).view(np.float32).tolist()

Defensive patterns

Strategy: validation

Validate before calling

def check_bfloat16_vector(data: bytes, dim: int) -> bool:
    return len(data) > 0 and len(data) == dim * 2

Type guard

def is_valid_bfloat16_blob(bytes_data: bytes, dim: int) -> bool:
    """True when bytes_data holds dim bfloat16 (truncated float32) values."""
    return isinstance(bytes_data, (bytes, bytearray)) and len(bytes_data) == dim * 2

Prevention

Validate len == dim*2 before decoding.
For correct values decode bfloat16 by bit-shifting: (u16 << 16).view(float32); the built-in normalization is a placeholder.
Confirm the column type label matches the actual encoding (bf16 vs f16 vs f32) before analysis.

When it happens

Trigger: Explicit dim mismatch; truncated blob; feeding float32 or float16 bytes while declaring BFloat16Vector.

Common situations: Index/vector type changed after files were written; mislabeled column; incomplete export.

Related errors

AI-assisted analysis of milvus-io/milvus@b43a76673a (2026-08-15). Data as JSON: /api/errors/1caef2b626b7d052. Report an issue: GitHub.