apache/druid · error · RuntimeException (Druid RE)

Failed to announce DiscoveryDruidNode[%s]

Error message

Failed to announce DiscoveryDruidNode[%s]

What it means

K8sDruidNodeAnnouncer.announce() writes the DiscoveryDruidNode JSON into the pod's own annotations via the K8s API (with retries, per the surrounding RetryUtils block). Any exception during that update — serialization of the node, ApiException from the API (RBAC/409 conflict/413 annotation size), or retry exhaustion — is wrapped in this RE identifying the node.

Source

Thrown at extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/K8sDruidNodeAnnouncer.java:122

      patches.add(createPatchObj(OP_ADD, getPodDefAnnocationPath(infoAnnotation), jsonMapper.writeValueAsString(discoveryDruidNode)));

      // Creating patch string outside of retry block to not retry json serialization failures
      String jsonPatchStr = jsonMapper.writeValueAsString(patches);
      LOGGER.info("Json Patch For Node Announcement: [%s]", jsonPatchStr);

      RetryUtils.retry(
          () -> {
            k8sApiClient.patchPod(podInfo.getPodName(), podInfo.getPodNamespace(), jsonPatchStr);
            return "na";
          },
          (throwable) -> true,
          3
      );

      LOGGER.info("Announced DiscoveryDruidNode[%s]", discoveryDruidNode);
    }
    catch (Exception ex) {
      throw new RE(ex, "Failed to announce DiscoveryDruidNode[%s]", discoveryDruidNode);
    }
  }

  @Override
  public void unannounce(DiscoveryDruidNode discoveryDruidNode)
  {
    LOGGER.info("Unannouncing DiscoveryDruidNode[%s]", discoveryDruidNode);

    String roleAnnouncementLabel = getRoleAnnouncementLabel(discoveryDruidNode.getNodeRole());
    String idHashAnnouncementLabel = getIdHashAnnouncementLabel();
    String clusterIdentifierAnnouncementLabel = getClusterIdentifierAnnouncementLabel();
    String infoAnnotation = getInfoAnnotation(discoveryDruidNode.getNodeRole());

    try {
      List<Map<String, Object>> patches = new ArrayList<>();
      patches.add(createPatchObj(OP_REMOVE, getPodDefLabelPath(roleAnnouncementLabel), null));
      patches.add(createPatchObj(OP_REMOVE, getPodDefLabelPath(idHashAnnouncementLabel), null));
      patches.add(createPatchObj(OP_REMOVE, getPodDefLabelPath(clusterIdentifierAnnouncementLabel), null));

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Read the cause: 403 → grant the service account patch/update on pods; 413/size → reduce node payload; 409 → check for concurrent announcers.
2. Verify druid_discovery_k8s announcer config (namespace, pod name resolution) matches where the pod actually runs.
3. Ensure the K8s API server is reachable and the service-account token is valid at startup.
4. Check retry settings; if retries are exhausted due to persistent conflicts, ensure only one announcer updates the pod.

Example fix

// before: SA without pod patch rights => RE after retries
// after
kubectl create role druid-announcer --verb=get,patch,update --resource=pods -n druid
kubectl create rolebinding druid-announcer --role=druid-announcer --serviceaccount=druid:druid-sa -n druid

kubectl auth can-i patch pods -n <namespace>   # must be yes for the Druid SA
# check annotation headroom
kubectl get pod <pod> -o jsonpath='{.metadata.annotations}' | wc -c  # far below 256KB

try {
  announcer.announce(discoveryDruidNode);
} catch (RE e) {
  Throwable cause = e.getCause();
  if (cause instanceof ApiException) {
    int code = ((ApiException) cause).getCode();
    if (code == 403) fixRbac();          // patch pods
    else if (code == 413) shrinkPayload(); // annotation too large
    else retryWithBackoff();
  }
}

When it happens

Trigger: announce(DiscoveryDruidNode) failing after retries: patch/update of the pod's druid.information.<role> annotation rejected by the K8s API, or the node JSON cannot be serialized.

Common situations: Service account lacking patch permission on pods; annotation exceeds K8s' 256KB total annotation limit; too-large DiscoveryDruidNode payloads (many services); optimistic-concurrency 409 conflicts; API server unreachable at process start.

Understand the failure class

Background: "API error: {status}" and "HTTP 401/403/404/429/5xx" errors: non-2xx HTTP responses explained — this error's family across 27 libraries.

Related errors

• Expection in watching pods, code[%d] and error[%s].
• Failed to get current leader for [%s]
• Failed to patch pod[%s/%s], code[%d], error[%s].
• Expection in listing pods, code[%d] and error[%s].
• Failed to deserialize DiscoveryDruidNode[%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/ce353eb72d3c5fa9. Report an issue: GitHub.