{"record":{"id":"ce353eb72d3c5fa9","repo":"apache/druid","slug":"failed-to-announce-discoverydruidnode-s","errorCode":null,"errorMessage":"Failed to announce DiscoveryDruidNode[%s]","messagePattern":"Failed to announce DiscoveryDruidNode\\[(.+?)\\]","errorType":"exception","errorClass":"RuntimeException (Druid RE)","httpStatus":null,"severity":"error","filePath":"extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java","lineNumber":122,"sourceCode":"      patches.add(createPatchObj(OP_ADD, getPodDefAnnocationPath(infoAnnotation), jsonMapper.writeValueAsString(discoveryDruidNode)));\n\n      // Creating patch string outside of retry block to not retry json serialization failures\n      String jsonPatchStr = jsonMapper.writeValueAsString(patches);\n      LOGGER.info(\"Json Patch For Node Announcement: [%s]\", jsonPatchStr);\n\n      RetryUtils.retry(\n          () -> {\n            k8sApiClient.patchPod(podInfo.getPodName(), podInfo.getPodNamespace(), jsonPatchStr);\n            return \"na\";\n          },\n          (throwable) -> true,\n          3\n      );\n\n      LOGGER.info(\"Announced DiscoveryDruidNode[%s]\", discoveryDruidNode);\n    }\n    catch (Exception ex) {\n      throw new RE(ex, \"Failed to announce DiscoveryDruidNode[%s]\", discoveryDruidNode);\n    }\n  }\n\n  @Override\n  public void unannounce(DiscoveryDruidNode discoveryDruidNode)\n  {\n    LOGGER.info(\"Unannouncing DiscoveryDruidNode[%s]\", discoveryDruidNode);\n\n    String roleAnnouncementLabel = getRoleAnnouncementLabel(discoveryDruidNode.getNodeRole());\n    String idHashAnnouncementLabel = getIdHashAnnouncementLabel();\n    String clusterIdentifierAnnouncementLabel = getClusterIdentifierAnnouncementLabel();\n    String infoAnnotation = getInfoAnnotation(discoveryDruidNode.getNodeRole());\n\n    try {\n      List<Map<String, Object>> patches = new ArrayList<>();\n      patches.add(createPatchObj(OP_REMOVE, getPodDefLabelPath(roleAnnouncementLabel), null));\n      patches.add(createPatchObj(OP_REMOVE, getPodDefLabelPath(idHashAnnouncementLabel), null));\n      patches.add(createPatchObj(OP_REMOVE, getPodDefLabelPath(clusterIdentifierAnnouncementLabel), null));","sourceCodeStart":104,"sourceCodeEnd":140,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java#L104-L140","documentation":"K8sDruidNodeAnnouncer.announce() writes the DiscoveryDruidNode JSON into the pod's own annotations via the K8s API (with retries, per the surrounding RetryUtils block). Any exception during that update — serialization of the node, ApiException from the API (RBAC/409 conflict/413 annotation size), or retry exhaustion — is wrapped in this RE identifying the node.","triggerScenarios":"announce(DiscoveryDruidNode) failing after retries: patch/update of the pod's druid.information.<role> annotation rejected by the K8s API, or the node JSON cannot be serialized.","commonSituations":"Service account lacking patch permission on pods; annotation exceeds K8s' 256KB total annotation limit; too-large DiscoveryDruidNode payloads (many services); optimistic-concurrency 409 conflicts; API server unreachable at process start.","solutions":["Read the cause: 403 → grant the service account patch/update on pods; 413/size → reduce node payload; 409 → check for concurrent announcers.","Verify druid_discovery_k8s announcer config (namespace, pod name resolution) matches where the pod actually runs.","Ensure the K8s API server is reachable and the service-account token is valid at startup.","Check retry settings; if retries are exhausted due to persistent conflicts, ensure only one announcer updates the pod."],"exampleFix":"// before: SA without pod patch rights => RE after retries\n// after\nkubectl create role druid-announcer --verb=get,patch,update --resource=pods -n druid\nkubectl create rolebinding druid-announcer --role=druid-announcer --serviceaccount=druid:druid-sa -n druid","handlingStrategy":"retry","validationCode":"kubectl auth can-i patch pods -n <namespace>   # must be yes for the Druid SA\n# check annotation headroom\nkubectl get pod <pod> -o jsonpath='{.metadata.annotations}' | wc -c  # far below 256KB","typeGuard":null,"tryCatchPattern":"try {\n  announcer.announce(discoveryDruidNode);\n} catch (RE e) {\n  Throwable cause = e.getCause();\n  if (cause instanceof ApiException) {\n    int code = ((ApiException) cause).getCode();\n    if (code == 403) fixRbac();          // patch pods\n    else if (code == 413) shrinkPayload(); // annotation too large\n    else retryWithBackoff();\n  }\n}","preventionTips":["Grant get/patch/update on pods to the Druid service account.","Keep DiscoveryDruidNode payloads small (services list) to stay under K8s annotation size limits.","Ensure only one announcer per pod updates annotations to avoid 409 conflicts.","Verify API-server reachability and token validity at startup; announce failures during shutdown are often transient."],"tags":["kubernetes","announcer","apiexception","rbac","annotations"],"backgroundTag":"http-error-response","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}