{"record":{"id":"ac0d950b5d286ae4","repo":"apache/cassandra","slug":"failed-to-incrementally-build-indexes","errorCode":null,"errorMessage":"Failed to incrementally build indexes {}","messagePattern":"Failed to incrementally build indexes (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/index/SecondaryIndexManager.java","lineNumber":623,"sourceCode":"        Map<Index.IndexBuildingSupport, Set<Index>> byType = new HashMap<>();\n        for (Index index : toBuild)\n        {\n            Set<Index> stored = byType.computeIfAbsent(index.getBuildTaskSupport(), i -> new HashSet<>());\n            stored.add(index);\n        }\n\n        // Schedule all index building tasks with callbacks to handle success and failure\n        List<Future<?>> futures = new ArrayList<>(byType.size());\n        byType.forEach((buildingSupport, groupedIndexes) ->\n        {\n            SecondaryIndexBuilder builder = buildingSupport.getIndexBuildTask(baseCfs, groupedIndexes, sstables, false);\n            AsyncPromise<Object> build = new AsyncPromise<>();\n            CompactionManager.instance.submitIndexBuild(builder).addCallback(new FutureCallback<Object>()\n            {\n                @Override\n                public void onFailure(Throwable t)\n                {\n                    logger.warn(\"Failed to incrementally build indexes {}\", getIndexNames(groupedIndexes));\n                    build.tryFailure(t);\n                }\n\n                @Override\n                public void onSuccess(Object o)\n                {\n                    logger.info(\"Incremental index build of {} completed\", getIndexNames(groupedIndexes));\n                    build.trySuccess(o);\n                }\n            });\n            futures.add(build);\n        });\n\n        // Finally wait for the index builds to finish\n        FBUtilities.waitOnFutures(futures);\n    }\n\n    /**","sourceCodeStart":605,"sourceCodeEnd":641,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/index/SecondaryIndexManager.java#L605-L641","documentation":"When indexes with isBuilt=false or isIndexed=true are built incrementally, SecondaryIndexManager submits an index build to CompactionManager and attaches a FutureCallback. On failure of that background compaction-driven build, the callback logs this WARN and propagates the failure into the `build` promise so the composed future (and callers like buildIndexesBlocking) see the error. The index is subsequently marked failed.","triggerScenarios":"submitIndexBuild future completing exceptionally — e.g. the underlying SSTable being compacted away mid-build, an IOException while reading columns during the index build, or a shutdown interrupt cancelling the compaction task.","commonSituations":"Adding a new secondary index while heavy compaction is running; node decommission/shutdown interrupting an index build; SAI or custom index throwing during scan of memtable/SSTable data.","solutions":["Check the propagated Throwable on the returned future / subsequent 'Index build of {} failed' log to find the root cause.","Run nodetool rebuild_index (or REBUILD INDEX) on the affected indexes to retry from scratch.","If caused by concurrent compaction, retry after compaction pressure subsides.","Ensure sufficient disk space and that the node was not shut down mid-build."],"exampleFix":"// before: build failure only warned, caller must notice\nlogger.warn(\"Failed to incrementally build indexes {}\", getIndexNames(groupedIndexes));\nbuild.tryFailure(t);\n// after: react to the composed future to trigger a rebuild\nbuild.addListener(() -> { if (!build.isSuccess()) logger.error(\"Scheduling rebuild for {}\", getIndexNames(groupedIndexes)); });","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { buildResult.get(30, TimeUnit.MINUTES); } catch (ExecutionException e) { logger.error(\"index build failed, scheduling rebuild_index\", e.getCause()); }","preventionTips":["Avoid creating indexes during heavy compaction windows.","Ensure enough disk headroom for both compaction and index build output.","Monitor background build futures and automate rebuild_index on failure.","Don't stop nodes mid-build; use nodetool compactionstats to check in-flight index builds."],"tags":["index","compaction","background-build","future"],"backgroundTag":"database-query-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}