{"record":{"id":"30769b6f53b5e3bd","repo":"elastic/elasticsearch","slug":"failed-to-merge-gpu-index","errorCode":null,"errorMessage":"Failed to merge GPU index: ","messagePattern":"Failed to merge GPU index: ","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java","lineNumber":659,"sourceCode":"                    // we just build a mock graph where every node is connected to every other node\n                    generateMockGraphAndWriteMeta(fieldInfo, numVectors);\n                } else if (dataType == CuVSMatrix.DataType.FLOAT) {\n                    mergeFloatVectorField(fieldInfo, mergeState, floatVectorValues, numVectors);\n                } else {\n                    // During merging, we use quantized data, so we need to support byte[] too.\n                    // That's how our current formats work: use floats during indexing, and quantized data to build a graph\n                    // during merging.\n                    assert dataType == CuVSMatrix.DataType.BYTE;\n                    assert flatVectorsReader instanceof QuantizedVectorsReader;\n                    BaseQuantizedByteVectorValues byteVectorValues = ((QuantizedVectorsReader) flatVectorsReader).getQuantizedVectorValues(\n                        fieldInfo.name\n                    );\n                    mergeByteVectorField(fieldInfo, mergeState, byteVectorValues, numVectors);\n                }\n                var elapsed = System.nanoTime() - started;\n                logger.debug(\"Merged [{}] vectors in [{}ms]\", numVectors, elapsed / 1_000_000.0);\n            } catch (Throwable t) {\n                throw new IOException(\"Failed to merge GPU index: \", t);\n            }\n        };\n    }\n\n    private void mergeByteVectorField(\n        FieldInfo fieldInfo,\n        MergeState mergeState,\n        BaseQuantizedByteVectorValues byteVectorValues,\n        int numVectors\n    ) throws IOException, InterruptedException {\n        CagraIndexParams cagraIndexParams = createCagraIndexParams(\n            fieldInfo.getVectorSimilarityFunction(),\n            numVectors,\n            fieldInfo.getVectorDimension()\n        );\n\n        IndexInput slice = byteVectorValues.getSlice();\n        var input = slice == null ? null : FilterIndexInput.unwrap(slice);","sourceCodeStart":641,"sourceCodeEnd":677,"githubUrl":"https://github.com/elastic/elasticsearch/blob/db6a809a667c081ca1dc7500389d26975573215f/libs/gpu-codec/src/main/java/org/elasticsearch/gpu/codec/ES92GpuHnswVectorsWriter.java#L641-L677","documentation":"Thrown by the lambda returned from ES92GpuHnswVectorsWriter.mergeOneField() when any Throwable occurs during the GPU-accelerated segment merge. This wraps all exceptions from the merge pipeline: opening the flat reader, reading float/byte vectors, acquiring GPU resources, building the CAGRA index, and writing the merged graph. The exception is always wrapped in an IOException regardless of the original type.","triggerScenarios":"Failure during Lucene segment merging when the GPU codec is active. This includes: flat reader open failures, vector iteration failures, GPU resource acquisition failures, CAGRA index build failures during merge, MemorySegment mapping failures for merged data, or quantized vector reading failures (for int8_hnsw fields).","commonSituations":"Large segment merges exceeding GPU memory; corrupted merged vector data; quantized vector reader returning unexpected data during BYTE merge path; concurrent merge threads competing for GPU resources; disk I/O failures during temp file creation for large merged segments.","solutions":["Inspect IOException.getCause() for the root exception.","If GPU resource contention, reduce index.merge.scheduler.max_thread_count or the number of concurrent merges.","If the cause is in the BYTE/quantized merge path, verify that QuantizedVectorsReader returns valid data (the assert at line 650 checks flatVectorsReader instanceof QuantizedVectorsReader).","For MemorySegment mapping failures in the merge path, check that the merged segment's directory is an FSDirectory (see error 612).","Consider disabling the GPU codec for problematic indices as a diagnostic step."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    IORunnable mergeTask = writer.mergeOneField(fieldInfo, mergeState);\n    mergeTask.run();\n} catch (IOException e) {\n    Throwable cause = e.getCause();\n    logger.warn(\"GPU merge failed for field {}, falling back or investigating: {}\", fieldInfo.name, cause);\n    throw e;\n}","preventionTips":["Limit concurrent merges to avoid GPU resource contention.","Ensure the merged segment directory is an FSDirectory for mmap fallback.","Verify quantized vector readers return valid data for BYTE merge path.","Monitor GPU memory during large segment merges."],"tags":["gpu-codec","vector-search","merge","gpu","cuda","ioexception-wrapper"],"backgroundTag":null,"analyzedSha":"db6a809a667c081ca1dc7500389d26975573215f","analyzedAt":"2026-08-12T01:39:14.192Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}