elastic/elasticsearch · critical · IOException

Failed to flush GPU index:

Error message

Failed to flush GPU index: 

What it means

Thrown by ES92GpuHnswVectorsWriter.flush() when any non-IOException Throwable (including RuntimeException, Error, or native GPU errors) occurs during flushFieldsWithoutMemoryMappedFile(). The original exception is wrapped as the cause of an IOException. This is a catch-all that ensures the flush operation always produces an IOException for the Lucene segment writer framework.

Source

Thrown at libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java:179

    /**
     * Flushes vector data and associated data to disk.
     * <p>
     * This method and the private helpers it calls only need to support FLOAT32.
     * For FlatFieldVectorWriter we only need to support float[] during flush: during indexing users provide floats[], and pass floats to
     * FlatFieldVectorWriter, even when we have a BYTE dataType (i.e. an "int8_hnsw" type).
     * During merging, we use quantized data, so we need to support byte[] too (see {@link ES92GpuHnswVectorsWriter#mergeOneField}),
     * but not here.
     * That's how our other current formats work: use floats during indexing, and quantized data to build graph during merging.
     * </p>
     */
    @Override
    public void flush(int maxDoc, Sorter.DocMap sortMap) throws IOException {
        var started = System.nanoTime();
        flatVectorWriter.flush(maxDoc, sortMap);
        try {
            flushFieldsWithoutMemoryMappedFile(sortMap);
        } catch (Throwable t) {
            throw new IOException("Failed to flush GPU index: ", t);
        }
        var elapsed = System.nanoTime() - started;
        logger.debug("Flush total time [{}ms]", elapsed / 1_000_000.0);
    }

    private void flushFieldsWithoutMemoryMappedFile(Sorter.DocMap sortMap) throws IOException {
        // No tmp file written, or the file cannot be mmapped
        for (FieldWriter field : fields) {
            var started = System.nanoTime();
            var fieldInfo = field.fieldInfo;
            int dims = fieldInfo.getVectorDimension();

            var originalVectors = field.flatFieldVectorsWriter.getVectors();
            final List<float[]> vectorsInSortedOrder = sortMap == null
                ? originalVectors
                : getVectorsInSortedOrder(field, sortMap, originalVectors);
            int numVectors = vectorsInSortedOrder.size();
            CagraIndexParams cagraIndexParams = createCagraIndexParams(fieldInfo.getVectorSimilarityFunction(), numVectors, dims);

View on GitHub (pinned to db6a809a66)

Solutions

1. Examine the wrapped cause (IOException.getCause()) for the root error — it will be a native GPU error, CUDA exception, or RuntimeException.
2. If the cause is GPU OOM, reduce the segment size or increase GPU memory, or let the system fall back to CPU (the code does this automatically when cuVSResourceManager.tryAcquire returns null).
3. Verify CUDA/CuVS library compatibility with the installed GPU driver.
4. Check available disk space for the vector index output files.
5. If the cause is a native crash, check dmesg/system logs for GPU error messages.

try {
    writer.flush(maxDoc, sortMap);
} catch (IOException e) {
    Throwable cause = e.getCause();
    if (cause != null) {
        logger.error("GPU flush failed due to: " + cause.getClass().getSimpleName(), cause);
    }
    throw e;
}

When it happens

Trigger: Any failure during the GPU graph build and metadata write phase of segment flush. This includes: GPU resource acquisition failure, CAGRA index build failure (CUDA errors, out of GPU memory), MemorySegment mapping failures, CuVS native library errors, or bugs in graph serialization. Also fires when the CPU fallback path (buildCpuGraphAndWriteMeta) fails.

Common situations: GPU driver/CUDA version mismatch causing native CAGRA errors; GPU out of memory when building large indexes; CuVS library not properly initialized; corrupted vector data causing native code to fail; disk full when writing the vector index files.

Related errors

• Failed to merge GPU index:
• Failed to write GPU index:
• Dataset dimensions must be positive: rows={}, features={}
• input cannot be null
• segment of size [{}] too small for expected {} float vectors

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/609f77cab1731e21. Report an issue: GitHub.