elastic/elasticsearch · critical · IOException

Failed to merge GPU index:

Error message

Failed to merge GPU index: 

What it means

Thrown by the lambda returned from ES92GpuHnswVectorsWriter.mergeOneField() when any Throwable occurs during the GPU-accelerated segment merge. This wraps all exceptions from the merge pipeline: opening the flat reader, reading float/byte vectors, acquiring GPU resources, building the CAGRA index, and writing the merged graph. The exception is always wrapped in an IOException regardless of the original type.

Source

Thrown at libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java:659

                    // we just build a mock graph where every node is connected to every other node
                    generateMockGraphAndWriteMeta(fieldInfo, numVectors);
                } else if (dataType == CuVSMatrix.DataType.FLOAT) {
                    mergeFloatVectorField(fieldInfo, mergeState, floatVectorValues, numVectors);
                } else {
                    // During merging, we use quantized data, so we need to support byte[] too.
                    // That's how our current formats work: use floats during indexing, and quantized data to build a graph
                    // during merging.
                    assert dataType == CuVSMatrix.DataType.BYTE;
                    assert flatVectorsReader instanceof QuantizedVectorsReader;
                    BaseQuantizedByteVectorValues byteVectorValues = ((QuantizedVectorsReader) flatVectorsReader).getQuantizedVectorValues(
                        fieldInfo.name
                    );
                    mergeByteVectorField(fieldInfo, mergeState, byteVectorValues, numVectors);
                }
                var elapsed = System.nanoTime() - started;
                logger.debug("Merged [{}] vectors in [{}ms]", numVectors, elapsed / 1_000_000.0);
            } catch (Throwable t) {
                throw new IOException("Failed to merge GPU index: ", t);
            }
        };
    }

    private void mergeByteVectorField(
        FieldInfo fieldInfo,
        MergeState mergeState,
        BaseQuantizedByteVectorValues byteVectorValues,
        int numVectors
    ) throws IOException, InterruptedException {
        CagraIndexParams cagraIndexParams = createCagraIndexParams(
            fieldInfo.getVectorSimilarityFunction(),
            numVectors,
            fieldInfo.getVectorDimension()
        );

        IndexInput slice = byteVectorValues.getSlice();
        var input = slice == null ? null : FilterIndexInput.unwrap(slice);

View on GitHub (pinned to db6a809a66)

Solutions

1. Inspect IOException.getCause() for the root exception.
2. If GPU resource contention, reduce index.merge.scheduler.max_thread_count or the number of concurrent merges.
3. If the cause is in the BYTE/quantized merge path, verify that QuantizedVectorsReader returns valid data (the assert at line 650 checks flatVectorsReader instanceof QuantizedVectorsReader).
4. For MemorySegment mapping failures in the merge path, check that the merged segment's directory is an FSDirectory (see error 612).
5. Consider disabling the GPU codec for problematic indices as a diagnostic step.

try {
    IORunnable mergeTask = writer.mergeOneField(fieldInfo, mergeState);
    mergeTask.run();
} catch (IOException e) {
    Throwable cause = e.getCause();
    logger.warn("GPU merge failed for field {}, falling back or investigating: {}", fieldInfo.name, cause);
    throw e;
}

When it happens

Trigger: Failure during Lucene segment merging when the GPU codec is active. This includes: flat reader open failures, vector iteration failures, GPU resource acquisition failures, CAGRA index build failures during merge, MemorySegment mapping failures for merged data, or quantized vector reading failures (for int8_hnsw fields).

Common situations: Large segment merges exceeding GPU memory; corrupted merged vector data; quantized vector reader returning unexpected data during BYTE merge path; concurrent merge threads competing for GPU resources; disk I/O failures during temp file creation for large merged segments.

Related errors

• Failed to flush GPU index:
• Failed to write GPU index:
• Dataset dimensions must be positive: rows={}, features={}
• input cannot be null
• segment of size [{}] too small for expected {} float vectors

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/30769b6f53b5e3bd. Report an issue: GitHub.