{"record":{"id":"0528ef555717b0a0","repo":"apache/druid","slug":"failed-to-check-missing-segments-due-to-missing-re","errorCode":null,"errorMessage":"Failed to check missing segments due to missing responses from [%d] servers","messagePattern":"Failed to check missing segments due to missing responses from \\[(.+?)\\] servers","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/query/RetryQueryRunner.java","lineNumber":156,"sourceCode":"  private List<SegmentDescriptor> getMissingSegments(QueryPlus<T> queryPlus, final ResponseContext context)\n  {\n    // Sanity check before retrieving missingSegments from responseContext.\n    // The missingSegments in the responseContext is only valid when all servers have responded to the broker.\n    // The remainingResponses MUST be not null but 0 in the responseContext at this point.\n    final ConcurrentHashMap<String, Integer> idToRemainingResponses =\n        Preconditions.checkNotNull(\n            context.getRemainingResponses(),\n            \"%s in responseContext\",\n            Keys.REMAINING_RESPONSES_FROM_QUERY_SERVERS.getName()\n        );\n\n    final int remainingResponses = Preconditions.checkNotNull(\n        idToRemainingResponses.get(queryPlus.getQuery().getMostSpecificId()),\n        \"Number of remaining responses for query[%s]\",\n        queryPlus.getQuery().getMostSpecificId()\n    );\n    if (remainingResponses > 0) {\n      throw new ISE(\"Failed to check missing segments due to missing responses from [%d] servers\", remainingResponses);\n    }\n\n    // TODO: the sender's response may contain a truncated list of missing segments.\n    // Truncation is aggregated in the response context given as a parameter.\n    // Check the getTruncated() value: if true, then the we don't know the full set of\n    // missing segments.\n    final List<SegmentDescriptor> maybeMissingSegments = context.getMissingSegments();\n    if (maybeMissingSegments == null) {\n      return Collections.emptyList();\n    }\n\n    return jsonMapper.convertValue(\n        maybeMissingSegments,\n        new TypeReference<>() {}\n    );\n  }\n\n  /**","sourceCodeStart":138,"sourceCodeEnd":174,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/query/RetryQueryRunner.java#L138-L174","documentation":"RetryQueryRunner.getMissingSegments() aggregates per-server responses listing missing segments. Before building the retry request it asserts that every server responded (idToRemainingResponses count is 0). If any servers never replied, it throws IllegalStateException 'Failed to check missing segments due to missing responses from [N] servers', because retrying with incomplete knowledge could be wrong.","triggerScenarios":"Querying historical/realtime servers where some fail to return their missing-segment response (server down, timeout, dropped connection) so remainingResponses > 0 when the check runs.","commonSituations":"Cluster with dead or restarting historicals; network partitions during query; very short HTTP timeouts on fan-out requests; servers overloaded and dropping RPCs.","solutions":["Check cluster health: ensure all historicals serving the datasource are up and reachable.","Increase druid.request.timeout / query timeout so slow servers can respond.","Retry the query once transient servers recover; this is often a transient condition.","Investigate why specific servers did not respond (logs on broker/historical, network issues)."],"exampleFix":"// broker tuning\n// before\ndruid.request.timeout=PT30S\n// after\ndruid.request.timeout=PT5M","handlingStrategy":"retry","validationCode":"// before querying, verify expected historicals are serving the datasource\n// via Coordinator /druid/coordinator/v1/datasources/{ds}?full","typeGuard":null,"tryCatchPattern":"try {\n  sequence = retryRunner.run(queryPlus, ctx);\n} catch (IllegalStateException e) {\n  if (e.getMessage().startsWith(\"Failed to check missing segments\")) {\n    // backoff and re-issue the query\n  }\n}","preventionTips":["Monitor historical availability and replication","Keep druid.request.timeout generous enough for fan-out responses","Alert on servers failing to respond to broker RPCs"],"tags":["distributed","query","retry"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}