{"record":{"id":"772931e04f256182","repo":"SonarSource/sonarqube","slug":"distributed-cluster-action-timed-out-in-cluster-no","errorCode":null,"errorMessage":"Distributed cluster action timed out in cluster nodes ","messagePattern":"Distributed cluster action timed out in cluster nodes ","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/sonar-process/src/main/java/org/sonar/process/cluster/hz/DistributedAnswer.java","lineNumber":90,"sourceCode":"\n  public void setFailed(Member member, Exception e) {\n    failedMembers.put(member, e);\n  }\n\n  public void propagateExceptions() {\n    if (!failedMembers.isEmpty()) {\n      String failedMemberNames = failedMembers.keySet().stream()\n        .map(m -> m.getAttribute(NODE_NAME.getKey()))\n        .collect(Collectors.joining(\", \"));\n      throw new IllegalStateException(\"Distributed cluster action in cluster nodes \" + failedMemberNames + \" (other nodes may have timed out)\",\n        failedMembers.values().iterator().next());\n    }\n\n    if (!timedOutMembers.isEmpty()) {\n      String timedOutMemberNames = timedOutMembers.stream()\n        .map(m -> m.getAttribute(NODE_NAME.getKey()))\n        .collect(Collectors.joining(\", \"));\n      throw new IllegalStateException(\"Distributed cluster action timed out in cluster nodes \" + timedOutMemberNames);\n    }\n  }\n\n  /**\n   * Returns any answer. No guarantees are made on the order. Use this method if you only expect exactly one answer.\n   * @return the first answer, if any\n   */\n  public Optional<T> getSingleAnswer() {\n    return answers.values().stream().findFirst();\n  }\n}\n","sourceCodeStart":72,"sourceCodeEnd":102,"githubUrl":"https://github.com/SonarSource/sonarqube/blob/184c821202192afc1c599fc912d0889b69fffa53/server/sonar-process/src/main/java/org/sonar/process/cluster/hz/DistributedAnswer.java#L72-L102","documentation":"DistributedAnswer.propagateExceptions also handles timeouts: when no failures were returned but some members never answered (timedOutMembers non-empty), it throws IllegalStateException listing the timed-out node names.","triggerScenarios":"A distributed cluster action whose future timed out on one or more members; propagateExceptions() is called after awaiting answers and finds only timeouts (no explicit failures).","commonSituations":"Slow or overloaded ES/app node in the SonarQube cluster; network partition delaying Hazelcast responses; timeout configured too low for a heavyweight operation.","solutions":["Check the listed nodes' health, CPU, and network connectivity; restart unresponsive nodes.","Increase the distributed-operation timeout if operations are legitimately long-running.","Verify Hazelcast ports (sonar.cluster.hz.port range) are open between all members and no firewall drops packets."],"exampleFix":"// before\nsonar.cluster.requestwebservice.timeout=10000\n// after\nsonar.cluster.requestwebservice.timeout=60000  // or fix the slow node first","handlingStrategy":"retry","validationCode":"for (Member m : cluster.getMembers()) {\n  if (!m.getAddress().getInetSocketAddress().isResolved() || !ping(m)) {\n    throw new IllegalStateException(\"Member unreachable before distributed action: \" + m.getAttribute(\"name\"));\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  distributedAnswer.propagateExceptions();\n} catch (IllegalStateException e) {\n  logger.warn(\"Timed-out nodes: {}\", e.getMessage());\n  // backoff and retry once after checking node liveness\n}","preventionTips":["Size the Hazelcast operation timeout for your slowest node, not the average.","Alert on node CPU/memory saturation before timeouts occur.","Open Hazelcast port ranges on all firewalls between cluster members."],"tags":["cluster","timeout","hazelcast"],"backgroundTag":"request-timeout","analyzedSha":"184c821202192afc1c599fc912d0889b69fffa53","analyzedAt":"2026-09-09T12:23:51.573Z","contentChangedAt":"2026-09-09T12:23:51.573Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}