{"record":{"id":"dafb3673ce74108f","repo":"elastic/elasticsearch","slug":"segment-of-size-too-small-for-expected-flo","errorCode":null,"errorMessage":"segment of size [{}] too small for expected {} float vectors of {} dims","messagePattern":"segment of size \\[(.+?)\\] too small for expected (.+?) float vectors of (.+?) dims","errorType":"validation","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/DatasetUtilsImpl.java","lineNumber":61,"sourceCode":"\n    @Override\n    public CuVSMatrix fromInput(MemorySegment input, int numVectors, int dims, CuVSMatrix.DataType dataType) {\n        if (input == null) {\n            throw new IllegalArgumentException(\"input cannot be null\");\n        }\n        if (numVectors < 0 || dims < 0) {\n            throwIllegalArgumentException(numVectors, dims);\n        }\n        final int byteSize = dataType == CuVSMatrix.DataType.FLOAT ? Float.BYTES : Byte.BYTES;\n        if (((long) numVectors * dims * byteSize) > input.byteSize()) {\n            throwIllegalArgumentException(input, numVectors, dims);\n        }\n        return fromMemorySegment(input, numVectors, dims, dataType);\n    }\n\n    static void throwIllegalArgumentException(MemorySegment ms, int numVectors, int dims) {\n        var s = \"segment of size [\" + ms.byteSize() + \"] too small for expected \" + numVectors + \" float vectors of \" + dims + \" dims\";\n        throw new IllegalArgumentException(s);\n    }\n\n    static void throwIllegalArgumentException(int numVectors, int dims) {\n        String s;\n        if (numVectors < 0) {\n            s = \"negative number of vectors: \" + numVectors;\n        } else {\n            s = \"negative vector dims: \" + dims;\n        }\n        throw new IllegalArgumentException(s);\n    }\n}\n","sourceCodeStart":43,"sourceCodeEnd":74,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/DatasetUtilsImpl.java#L43-L74","documentation":"Thrown by DatasetUtilsImpl.fromInput() when the provided MemorySegment's byte size is smaller than what is required to hold the claimed number of vectors of the given dimensionality. The required size is computed as numVectors * dims * elementSize where elementSize is Float.BYTES (4) for FLOAT data or Byte.BYTES (1) for BYTE data. Note: the message says 'float vectors' even when DataType is BYTE, which is a cosmetic bug in the error text.","triggerScenarios":"Calling DatasetUtils.getInstance().fromInput(segment, numVectors, dims, dataType) where (long) numVectors * dims * byteSize > segment.byteSize(). Occurs when the MemorySegment was allocated for fewer vectors or fewer dimensions than the caller claims, or when a partially-written/truncated segment is passed. The internal callers are mergeFloatVectorField and mergeByteVectorField in ES92GpuHnswVectorsWriter, which pass mmap-backed segments from merged Lucene segments.","commonSituations":"Vector data file is corrupted or truncated after a crash; numVectors was computed from one source (e.g. FloatVectorValues.size()) but the underlying segment was written with different dimensions; the MemorySegment was sliced to the wrong length; an off-by-one in dimension calculation (e.g. sourceRowPitch vs packedRowSize mismatch in the BYTE/quantized path).","solutions":["Verify that numVectors matches the actual number of vectors stored in the MemorySegment — compare against the FloatVectorValues.size() or the segment metadata.","Check that dims matches fieldInfo.getVectorDimension() and that the segment was allocated with the same dimensionality.","Ensure the MemorySegment's byteSize() is at least numVectors * dims * (dataType == FLOAT ? 4 : 1).","If merging quantized (BYTE) data, verify that the packed row size passed to getContiguousPackedMemorySegment matches the actual data layout — the sourceRowPitch includes the 4-byte correction constant but packedRowSize does not.","Check for index corruption: run a lucene CheckIndex on the affected shard."],"exampleFix":"// before — dims mismatch between segment allocation and claim\nMemorySegment seg = arena.allocate((long) numVectors * (dims + 4)); // has padding\nCuVSMatrix m = DatasetUtils.getInstance().fromInput(seg, numVectors, dims, DataType.FLOAT);\n\n// after — segment size must match numVectors * dims * byteSize\nMemorySegment seg = arena.allocate((long) numVectors * dims * Float.BYTES);\nCuVSMatrix m = DatasetUtils.getInstance().fromInput(seg, numVectors, dims, DataType.FLOAT);","handlingStrategy":"validation","validationCode":"// Validate segment size before calling fromInput\nlong requiredBytes = (long) numVectors * dims * (dataType == CuVSMatrix.DataType.FLOAT ? Float.BYTES : Byte.BYTES);\nif (requiredBytes > segment.byteSize()) {\n    throw new IllegalStateException(String.format(\n        \"Insufficient segment: need %d bytes for %d vectors x %d dims, have %d\",\n        requiredBytes, numVectors, dims, segment.byteSize()));\n}\nCuVSMatrix m = DatasetUtils.getInstance().fromInput(segment, numVectors, dims, dataType);","typeGuard":null,"tryCatchPattern":"try {\n    CuVSMatrix m = DatasetUtils.getInstance().fromInput(segment, numVectors, dims, dataType);\n} catch (IllegalArgumentException e) {\n    // log segment.byteSize(), numVectors, dims for diagnosis\n    throw new IOException(\"Vector data segment validation failed\", e);\n}","preventionTips":["Always compute segment size as numVectors * dims * elementSize before allocation.","Verify numVectors comes from an authoritative source (e.g. FloatVectorValues.size()) and dims from fieldInfo.getVectorDimension().","For quantized (BYTE) data, ensure packedRowSize (not sourceRowPitch) is used when calling fromInput."],"tags":["gpu-codec","vector-search","validation","memory-segment","cuvs"],"backgroundTag":null,"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-13T09:17:06.757Z"}