apache/druid · error · IllegalStateException

Null queryRunner! Looks to be some segment unmapping action

Error message

Null queryRunner! Looks to be some segment unmapping action happening

What it means

GroupByMergingQueryRunner builds per-segment futures via a Function applied to query runners obtained from the timeline. If the runner Function yields null — meaning the segment's runner disappeared (segment unmapped/closed) while the query was being fanned out — it throws an ISE naming segment unmapping as the cause.

Source

Thrown at processing/src/main/java/org/apache/druid/query/groupby/epinephelinae/GroupByMergingQueryRunner.java:250

                  );
              final Grouper<RowBasedKey> grouper = pair.lhs;
              final Accumulator<AggregateResult, ResultRow> accumulator = pair.rhs;
              grouper.init();

              final ReferenceCountingResourceHolder<Grouper<RowBasedKey>> grouperHolder =
                  ReferenceCountingResourceHolder.fromCloseable(grouper);
              resources.register(grouperHolder);

              List<ListenableFuture<AggregateResult>> futures = Lists.newArrayList(
                      Iterables.transform(
                          queryables,
                          new Function<>()
                          {
                            @Override
                            public ListenableFuture<AggregateResult> apply(final QueryRunner<ResultRow> input)
                            {
                              if (input == null) {
                                throw new ISE("Null queryRunner! Looks to be some segment unmapping action happening");
                              }

                              final AbstractPrioritizedQueryRunnerCallable<AggregateResult, ResultRow> callable = new AbstractPrioritizedQueryRunnerCallable<>(priority, input)
                              {
                                @Override
                                public AggregateResult call()
                                {
                                  try (
                                      // These variables are used to close releasers automatically.
                                      @SuppressWarnings("unused")
                                      Closeable bufferReleaser = mergeBufferHolder.increment();
                                      @SuppressWarnings("unused")
                                      Closeable grouperReleaser = grouperHolder.increment()
                                  ) {
                                    // Return true if OK, false if resources were exhausted.
                                    return input.run(queryPlusForRunners, responseContext)
                                                .accumulate(AggregateResult.ok(), accumulator);
                                  }

View on GitHub (pinned to 9b90983fd2)

Solutions

Retry the query — transient segment churn is expected in a Druid cluster
Inspect coordinator/historical logs for segment drop/load races at the failure time
Increase replica count so at least one replica serves the segment
Tune coordinator load/drop windows and balancing periods to reduce churn
Ensure server-level segment lifecycle code never returns null runners mid-query (guard in server tier)

Example fix

// before
ListenableFuture<AggregateResult> f = Futures.transform(runnerFn.apply(null-runner), ...);
// after
QueryRunner<ResultRow> input = runnerFn.apply(segmentId);
if (input == null) {
  throw new ResourceLimitException("Segment unavailable, retry query"); // retryable instead of ISE
}

Defensive patterns

Strategy: retry

Validate before calling

if (input == null) { throw new RetryQueryException("segment runner disappeared; retry"); }

Type guard

boolean hasRunner(Function<Object, QueryRunner<ResultRow>> fn, SegmentDescriptor d) { return fn.apply(d) != null; }

Try / catch

try { merge(query); } catch (ISE e) { if (e.getMessage().contains("Null queryRunner")) { retryWithBackoff(query); } }

Prevention

Retry broker queries on segment-unmapped style ISEs
Keep replicas >= 2 for query availability during segment churn
Reduce coordinator drop/balance aggressiveness under heavy query load

When it happens

Trigger: Query fan-out via FunctionalQueryableRunner/Timeline where a segment is dropped or closed on the historical server concurrently with query execution, so the Function<QueryRunner> apply receives input == null.

Common situations: Coordinator-triggered segment drops/balancing during queries; real-time task handoff; replica serving segments being cleaned up; cache unmapping under memory pressure.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/bc647f50e57b0eae. Report an issue: GitHub.