apache/druid · error · IllegalStateException

Failed to check missing segments due to missing responses fr

Error message

Failed to check missing segments due to missing responses from [%d] servers

What it means

RetryQueryRunner.getMissingSegments() aggregates per-server responses listing missing segments. Before building the retry request it asserts that every server responded (idToRemainingResponses count is 0). If any servers never replied, it throws IllegalStateException 'Failed to check missing segments due to missing responses from [N] servers', because retrying with incomplete knowledge could be wrong.

Source

Thrown at server/src/main/java/org/apache/druid/query/RetryQueryRunner.java:156

  private List<SegmentDescriptor> getMissingSegments(QueryPlus<T> queryPlus, final ResponseContext context)
  {
    // Sanity check before retrieving missingSegments from responseContext.
    // The missingSegments in the responseContext is only valid when all servers have responded to the broker.
    // The remainingResponses MUST be not null but 0 in the responseContext at this point.
    final ConcurrentHashMap<String, Integer> idToRemainingResponses =
        Preconditions.checkNotNull(
            context.getRemainingResponses(),
            "%s in responseContext",
            Keys.REMAINING_RESPONSES_FROM_QUERY_SERVERS.getName()
        );

    final int remainingResponses = Preconditions.checkNotNull(
        idToRemainingResponses.get(queryPlus.getQuery().getMostSpecificId()),
        "Number of remaining responses for query[%s]",
        queryPlus.getQuery().getMostSpecificId()
    );
    if (remainingResponses > 0) {
      throw new ISE("Failed to check missing segments due to missing responses from [%d] servers", remainingResponses);
    }

    // TODO: the sender's response may contain a truncated list of missing segments.
    // Truncation is aggregated in the response context given as a parameter.
    // Check the getTruncated() value: if true, then the we don't know the full set of
    // missing segments.
    final List<SegmentDescriptor> maybeMissingSegments = context.getMissingSegments();
    if (maybeMissingSegments == null) {
      return Collections.emptyList();
    }

    return jsonMapper.convertValue(
        maybeMissingSegments,
        new TypeReference<>() {}
    );
  }

  /**

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Check cluster health: ensure all historicals serving the datasource are up and reachable.
2. Increase druid.request.timeout / query timeout so slow servers can respond.
3. Retry the query once transient servers recover; this is often a transient condition.
4. Investigate why specific servers did not respond (logs on broker/historical, network issues).

Example fix

// broker tuning
// before
druid.request.timeout=PT30S
// after
druid.request.timeout=PT5M

// before querying, verify expected historicals are serving the datasource
// via Coordinator /druid/coordinator/v1/datasources/{ds}?full

try {
  sequence = retryRunner.run(queryPlus, ctx);
} catch (IllegalStateException e) {
  if (e.getMessage().startsWith("Failed to check missing segments")) {
    // backoff and re-issue the query
  }
}

When it happens

Trigger: Querying historical/realtime servers where some fail to return their missing-segment response (server down, timeout, dropped connection) so remainingResponses > 0 when the check runs.

Common situations: Cluster with dead or restarting historicals; network partitions during query; very short HTTP timeouts on fan-out requests; servers overloaded and dropping RPCs.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Query[%s] timed out.
• Recoverable exception
• Incompatible aggregator factories, [this] and [other]
• TooManyInputFiles
• Stage[%d] work orders not found

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0528ef555717b0a0. Report an issue: GitHub.