{"record":{"id":"9645cf125dd5533a","repo":"elastic/elasticsearch","slug":"failed-to-write-gpu-index","errorCode":null,"errorMessage":"Failed to write GPU index: ","messagePattern":"Failed to write GPU index: ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java","lineNumber":306,"sourceCode":"                            index = null;\n                            resourcesHolder.close();\n                            graph = writeGraph(hostGraph, graphLevelNodeOffsets);\n                        }\n                    } else {\n                        graph = writeGraph(deviceGraph, graphLevelNodeOffsets);\n                    }\n                }\n            } finally {\n                if (index != null) {\n                    index.close();\n                }\n            }\n            long vectorIndexLength = vectorIndex.getFilePointer() - vectorIndexOffset;\n            writeMeta(fieldInfo, vectorIndexOffset, vectorIndexLength, (int) dataset.size(), graph, graphLevelNodeOffsets);\n        } catch (IOException e) {\n            throw e;\n        } catch (Throwable t) {\n            throw new IOException(\"Failed to write GPU index: \", t);\n        }\n    }\n\n    private void generateMockGraphAndWriteMeta(FieldInfo fieldInfo, int datasetSize) throws IOException {\n        try {\n            long vectorIndexOffset = vectorIndex.getFilePointer();\n            int[][] graphLevelNodeOffsets = new int[1][];\n            final HnswGraph graph = writeMockGraph(datasetSize, graphLevelNodeOffsets);\n            long vectorIndexLength = vectorIndex.getFilePointer() - vectorIndexOffset;\n            writeMeta(fieldInfo, vectorIndexOffset, vectorIndexLength, datasetSize, graph, graphLevelNodeOffsets);\n        } catch (IOException e) {\n            throw e;\n        } catch (Throwable t) {\n            throw new IOException(\"Failed to write GPU index: \", t);\n        }\n    }\n\n    private CagraIndex buildGPUIndex(","sourceCodeStart":288,"sourceCodeEnd":324,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java#L288-L324","documentation":"Thrown by generateGpuGraphAndWriteMeta() when a non-IOException Throwable occurs during the GPU graph build and serialization path. IOExceptions are re-thrown as-is; everything else (RuntimeException, Error, native exceptions from CAGRA/CUDA) is wrapped into an IOException with this message. This covers the full GPU pipeline: index build, graph extraction, graph-to-host copy, and Lucene format serialization.","triggerScenarios":"Failure during CagraIndex.build() (GPU computation), index.getGraph() (graph extraction), deviceGraph.toHost() (device-to-host transfer), or writeGraph() (serialization). Common root causes: CUDA errors, GPU memory exhaustion, CAGRA algorithm failures (e.g. IVF_PQ parameter issues), assertion errors in graph structure.","commonSituations":"Building a GPU index on a dataset that triggers a CAGRA bug (e.g. unsupported dimensions for IVF_PQ PQ dim computation); GPU memory pressure during large index builds; transient CUDA errors; CAGRA version incompatibility with the CuVS library.","solutions":["Inspect IOException.getCause() for the specific GPU/CAGRA/CUDA error.","If the cause is GPU OOM, reduce segment merge factor or the number of concurrent GPU operations.","For IVF_PQ algorithm issues (large datasets), verify that CuVSIvfPqParamsFactory produces valid parameters for the given dims — some dimension counts are not factorable into valid PQ dims.","Check CUDA driver and CuVS library versions are compatible.","As a workaround, force CPU graph building by reducing MIN_NUM_VECTORS_FOR_GPU_BUILD or disabling GPU codec."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    // GPU index build operation\n} catch (IOException e) {\n    Throwable root = e.getCause() != null ? e.getCause() : e;\n    if (root instanceof OutOfMemoryError) {\n        // handle GPU OOM — reduce batch size or disable GPU codec\n    }\n    throw e;\n}","preventionTips":["Monitor GPU memory during large index builds.","Verify CAGRA parameters (especially IVF_PQ dims) are valid for the dataset dimensions.","Ensure CUDA driver and CuVS library versions are compatible.","Consider reducing concurrent GPU operations via CuVSResourceManager pool size."],"tags":["gpu-codec","vector-search","gpu-build","cuda","ioexception-wrapper"],"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T06:17:24.410Z"}