elastic/elasticsearch · error · IllegalArgumentException

segment of size [{}] too small for expected {} float vectors

Error message

segment of size [{}] too small for expected {} float vectors of {} dims

What it means

Thrown by DatasetUtilsImpl.fromInput() when the provided MemorySegment's byte size is smaller than what is required to hold the claimed number of vectors of the given dimensionality. The required size is computed as numVectors * dims * elementSize where elementSize is Float.BYTES (4) for FLOAT data or Byte.BYTES (1) for BYTE data. Note: the message says 'float vectors' even when DataType is BYTE, which is a cosmetic bug in the error text.

Source

Thrown at libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/DatasetUtilsImpl.java:61

    @Override
    public CuVSMatrix fromInput(MemorySegment input, int numVectors, int dims, CuVSMatrix.DataType dataType) {
        if (input == null) {
            throw new IllegalArgumentException("input cannot be null");
        }
        if (numVectors < 0 || dims < 0) {
            throwIllegalArgumentException(numVectors, dims);
        }
        final int byteSize = dataType == CuVSMatrix.DataType.FLOAT ? Float.BYTES : Byte.BYTES;
        if (((long) numVectors * dims * byteSize) > input.byteSize()) {
            throwIllegalArgumentException(input, numVectors, dims);
        }
        return fromMemorySegment(input, numVectors, dims, dataType);
    }

    static void throwIllegalArgumentException(MemorySegment ms, int numVectors, int dims) {
        var s = "segment of size [" + ms.byteSize() + "] too small for expected " + numVectors + " float vectors of " + dims + " dims";
        throw new IllegalArgumentException(s);
    }

    static void throwIllegalArgumentException(int numVectors, int dims) {
        String s;
        if (numVectors < 0) {
            s = "negative number of vectors: " + numVectors;
        } else {
            s = "negative vector dims: " + dims;
        }
        throw new IllegalArgumentException(s);
    }
}

View on GitHub (pinned to db6a809a66)

Solutions

1. Verify that numVectors matches the actual number of vectors stored in the MemorySegment — compare against the FloatVectorValues.size() or the segment metadata.
2. Check that dims matches fieldInfo.getVectorDimension() and that the segment was allocated with the same dimensionality.
3. Ensure the MemorySegment's byteSize() is at least numVectors * dims * (dataType == FLOAT ? 4 : 1).
4. If merging quantized (BYTE) data, verify that the packed row size passed to getContiguousPackedMemorySegment matches the actual data layout — the sourceRowPitch includes the 4-byte correction constant but packedRowSize does not.
5. Check for index corruption: run a lucene CheckIndex on the affected shard.

Example fix

// before — dims mismatch between segment allocation and claim
MemorySegment seg = arena.allocate((long) numVectors * (dims + 4)); // has padding
CuVSMatrix m = DatasetUtils.getInstance().fromInput(seg, numVectors, dims, DataType.FLOAT);

// after — segment size must match numVectors * dims * byteSize
MemorySegment seg = arena.allocate((long) numVectors * dims * Float.BYTES);
CuVSMatrix m = DatasetUtils.getInstance().fromInput(seg, numVectors, dims, DataType.FLOAT);

// Validate segment size before calling fromInput
long requiredBytes = (long) numVectors * dims * (dataType == CuVSMatrix.DataType.FLOAT ? Float.BYTES : Byte.BYTES);
if (requiredBytes > segment.byteSize()) {
    throw new IllegalStateException(String.format(
        "Insufficient segment: need %d bytes for %d vectors x %d dims, have %d",
        requiredBytes, numVectors, dims, segment.byteSize()));
}
CuVSMatrix m = DatasetUtils.getInstance().fromInput(segment, numVectors, dims, dataType);

try {
    CuVSMatrix m = DatasetUtils.getInstance().fromInput(segment, numVectors, dims, dataType);
} catch (IllegalArgumentException e) {
    // log segment.byteSize(), numVectors, dims for diagnosis
    throw new IOException("Vector data segment validation failed", e);
}

When it happens

Trigger: Calling DatasetUtils.getInstance().fromInput(segment, numVectors, dims, dataType) where (long) numVectors * dims * byteSize > segment.byteSize(). Occurs when the MemorySegment was allocated for fewer vectors or fewer dimensions than the caller claims, or when a partially-written/truncated segment is passed. The internal callers are mergeFloatVectorField and mergeByteVectorField in ES92GpuHnswVectorsWriter, which pass mmap-backed segments from merged Lucene segments.

Common situations: Vector data file is corrupted or truncated after a crash; numVectors was computed from one source (e.g. FloatVectorValues.size()) but the underlying segment was written with different dimensions; the MemorySegment was sliced to the wrong length; an off-by-one in dimension calculation (e.g. sourceRowPitch vs packedRowSize mismatch in the BYTE/quantized path).

Related errors

• negative number of vectors: {}
• negative vector dims: {}
• maxConn must be positive and less than or equal to 512; maxC
• beamWidth must be positive and less than or equal to 3200; b
• Field [{}] must have FLOAT32 encoding, got: {}

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/dafb3673ce74108f. Report an issue: GitHub.