elastic/elasticsearch · critical · IOException

Failed to write GPU index:

Error message

Failed to write GPU index: 

What it means

Thrown by generateGpuGraphAndWriteMeta() when a non-IOException Throwable occurs during the GPU graph build and serialization path. IOExceptions are re-thrown as-is; everything else (RuntimeException, Error, native exceptions from CAGRA/CUDA) is wrapped into an IOException with this message. This covers the full GPU pipeline: index build, graph extraction, graph-to-host copy, and Lucene format serialization.

Source

Thrown at libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java:306

                            index = null;
                            resourcesHolder.close();
                            graph = writeGraph(hostGraph, graphLevelNodeOffsets);
                        }
                    } else {
                        graph = writeGraph(deviceGraph, graphLevelNodeOffsets);
                    }
                }
            } finally {
                if (index != null) {
                    index.close();
                }
            }
            long vectorIndexLength = vectorIndex.getFilePointer() - vectorIndexOffset;
            writeMeta(fieldInfo, vectorIndexOffset, vectorIndexLength, (int) dataset.size(), graph, graphLevelNodeOffsets);
        } catch (IOException e) {
            throw e;
        } catch (Throwable t) {
            throw new IOException("Failed to write GPU index: ", t);
        }
    }

    private void generateMockGraphAndWriteMeta(FieldInfo fieldInfo, int datasetSize) throws IOException {
        try {
            long vectorIndexOffset = vectorIndex.getFilePointer();
            int[][] graphLevelNodeOffsets = new int[1][];
            final HnswGraph graph = writeMockGraph(datasetSize, graphLevelNodeOffsets);
            long vectorIndexLength = vectorIndex.getFilePointer() - vectorIndexOffset;
            writeMeta(fieldInfo, vectorIndexOffset, vectorIndexLength, datasetSize, graph, graphLevelNodeOffsets);
        } catch (IOException e) {
            throw e;
        } catch (Throwable t) {
            throw new IOException("Failed to write GPU index: ", t);
        }
    }

    private CagraIndex buildGPUIndex(

View on GitHub (pinned to db6a809a66)

Solutions

1. Inspect IOException.getCause() for the specific GPU/CAGRA/CUDA error.
2. If the cause is GPU OOM, reduce segment merge factor or the number of concurrent GPU operations.
3. For IVF_PQ algorithm issues (large datasets), verify that CuVSIvfPqParamsFactory produces valid parameters for the given dims — some dimension counts are not factorable into valid PQ dims.
4. Check CUDA driver and CuVS library versions are compatible.
5. As a workaround, force CPU graph building by reducing MIN_NUM_VECTORS_FOR_GPU_BUILD or disabling GPU codec.

try {
    // GPU index build operation
} catch (IOException e) {
    Throwable root = e.getCause() != null ? e.getCause() : e;
    if (root instanceof OutOfMemoryError) {
        // handle GPU OOM — reduce batch size or disable GPU codec
    }
    throw e;
}

When it happens

Trigger: Failure during CagraIndex.build() (GPU computation), index.getGraph() (graph extraction), deviceGraph.toHost() (device-to-host transfer), or writeGraph() (serialization). Common root causes: CUDA errors, GPU memory exhaustion, CAGRA algorithm failures (e.g. IVF_PQ parameter issues), assertion errors in graph structure.

Common situations: Building a GPU index on a dataset that triggers a CAGRA bug (e.g. unsupported dimensions for IVF_PQ PQ dim computation); GPU memory pressure during large index builds; transient CUDA errors; CAGRA version incompatibility with the CuVS library.

Related errors

• Failed to flush GPU index:
• Failed to merge GPU index:
• segment of size [{}] too small for expected {} float vectors
• negative number of vectors: {}
• negative vector dims: {}

AI-assisted analysis of elastic/elasticsearch@db6a809a66 (2026-08-12). Data as JSON: /api/errors/9645cf125dd5533a. Report an issue: GitHub.