{"record":{"id":"a64c0bf1ee51a0f5","repo":"apache/druid","slug":"null-queryrunner-looks-to-be-some-segment-unmappi","errorCode":null,"errorMessage":"Null queryRunner! Looks to be some segment unmapping action happening","messagePattern":"Null queryRunner! Looks to be some segment unmapping action happening","errorType":"exception","errorClass":"ISE","httpStatus":null,"severity":"warning","filePath":"processing/src/main/java/org/apache/druid/query/ChainedExecutionQueryRunner.java","lineNumber":101,"sourceCode":"    final QueryPlus<T> threadSafeQueryPlus = queryPlus.withoutThreadUnsafeState();\n\n    final QueryContext context = query.context();\n    final boolean usePerSegmentTimeout = context.usePerSegmentTimeout();\n    final long perSegmentTimeout = context.getPerSegmentTimeout();\n    return new BaseSequence<>(\n        new BaseSequence.IteratorMaker<>()\n        {\n          @Override\n          public Iterator<T> make()\n          {\n            // Make it a List<> to materialize all the values (so that it will submit everything to the executor)\n            List<ListenableFuture<Iterable<T>>> futures =\n                Lists.newArrayList(\n                    Iterables.transform(\n                        queryables,\n                        input -> {\n                          if (input == null) {\n                            throw new ISE(\"Null queryRunner! Looks to be some segment unmapping action happening\");\n                          }\n\n                          final AbstractPrioritizedQueryRunnerCallable<Iterable<T>, T> callable = new AbstractPrioritizedQueryRunnerCallable<>(\n                              priority,\n                              input\n                          )\n                          {\n                            @Override\n                            public Iterable<T> call()\n                            {\n                              try {\n                                Sequence<T> result = input.run(threadSafeQueryPlus, responseContext);\n                                if (result == null) {\n                                  throw new ISE(\"Got a null result! Segments are missing!\");\n                                }\n\n                                List<T> retVal = result.toList();\n                                if (retVal == null) {","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/query/ChainedExecutionQueryRunner.java#L83-L119","documentation":"In ChainedExecutionQueryRunner.make, each queryable contributed to the chain is null-checked before a callable is scheduled; a null runner means a segment was unmapped (closed and evicted) between when the segment list was built and when the query actually ran. Druid throws ISE instead of silently skipping data, since results would be incomplete.","triggerScenarios":"A Historical/Mergeable queryable becomes null in the queryables iterable passed to make(): a segment dropped or replaced by a compaction/rebalance task while its reference was being handed to the query chain, typically under concurrent segment load/drop or broker/historical segment-handoff races.","commonSituations":"Queries racing with segment dropping (killing datasource segments), overlord/compaction swapping segments, historicals restarting while brokers still route to them, or retention rules dropping segments mid-query; frequently seen in autoscaling or aggressively compacting clusters.","solutions":["Retry the query; Druid's failure usually resolves once segment unmapping settles and the broker re-plans against the new segment set.","Check if a kill/compaction/retention task ran concurrently (overlord logs) and reschedule such maintenance away from query-heavy periods or use the timeline's grace behavior.","Ensure brokers refresh their timelines promptly (tune druid.broker.cache / segment lifecycle, disable stale lookups) and that historicals aren't dropping segments under memory pressure (check druid.processing.numMergeBuffers and heap sizing).","If reproducible without any segment churn, file a bug with the broker/historical logs — a null runner in a stable timeline indicates an internal race."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n  return queryRunner.run(query, responseContext);\n} catch (ISE e) {\n  if (e.getMessage().contains(\"Null queryRunner\")) {\n    return retryWithBackoff(query, 2); // segment set likely re-planned\n  }\n  throw e;\n}","preventionTips":["Avoid running kill/compaction/retention tasks during peak query windows.","Size historical heaps and processing buffers so segments aren't unmapped under memory pressure.","Retry idempotent queries once on this error before surfacing it to users.","Watch broker timeline refresh logs around segment drops to correlate this error with segment churn."],"tags":["segments","race-condition","distributed-query","lifecycle"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}