apache/druid · error · RuntimeException (Druid RE)

Failed to deserialize DiscoveryDruidNode[%s]

Error message

Failed to deserialize DiscoveryDruidNode[%s]

What it means

Thrown by DefaultK8sApiClient when the JSON stored in a Druid pod's Kubernetes annotation cannot be deserialized into a DiscoveryDruidNode via Jackson. The k8s driver-based discovery reads node info from the annotation named by K8sDruidNodeAnnouncer.getInfoAnnotation(nodeRole); if that payload is malformed, missing required fields, or was written by an incompatible Druid version, Jackson throws JsonProcessingException which is wrapped in this RE with the offending JSON string.

Source

Thrown at extensions-core/kubernetes-extensions/src/main/java/org/apache/druid/k8s/discovery/DefaultK8sApiClient.java:161

  {
    if (pod.getStatus() == null) {
      return false;
    }
    List<V1ContainerStatus> containerStatuses = pod.getStatus().getContainerStatuses();
    if (containerStatuses == null || containerStatuses.isEmpty()) {
      return false;
    }
    return containerStatuses.stream().allMatch(cs -> Boolean.TRUE.equals(cs.getReady()));
  }

  private DiscoveryDruidNode getDiscoveryDruidNodeFromPodDef(NodeRole nodeRole, V1Pod podDef)
  {
    String jsonStr = podDef.getMetadata().getAnnotations().get(K8sDruidNodeAnnouncer.getInfoAnnotation(nodeRole));
    try {
      return jsonMapper.readValue(jsonStr, DiscoveryDruidNode.class);
    }
    catch (JsonProcessingException ex) {
      throw new RE(ex, "Failed to deserialize DiscoveryDruidNode[%s]", jsonStr);
    }
  }

  @Override
  public WatchResult watchPods(String namespace, String labelSelector, String lastKnownResourceVersion, NodeRole nodeRole)
  {
    try {
      Watch<V1Pod> watch =
          Watch.createWatch(
              realK8sClient,
              coreV1Api.listNamespacedPodCall(
                  namespace,
                  null,
                  true,
                  null,
                  null,
                  labelSelector,
                  null,

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Inspect the JSON string in the error message; validate it parses and matches DiscoveryDruidNode (druid_node role, host, port, etc.).
2. Verify all pods in the watched namespace/labelSelector run a compatible Druid version announcing the same annotation format.
3. Check the annotation key for the nodeRole matches what K8sDruidNodeAnnouncer writes (getInfoAnnotation(nodeRole)).
4. Delete/re-announce stale pods or fix the annotation contents, then retry discovery.

Example fix

// before: annotation written manually with wrong shape
kubectl annotate pod druid-historicals-0 druid.information.historical='{"host":"h","port":8083}'
// after: let Druid announce it, or match DiscoveryDruidNode schema incl. druidNode and serverDiscParams
kubectl annotate pod druid-historicals-0 druid.information.historical='{"druidNode":{"host":"h","port":8083,"plaintextPort":8083},"nodeType":"historical","services":{...}}'

String ann = pod.getMetadata() == null || pod.getMetadata().getAnnotations() == null ? null : pod.getMetadata().getAnnotations().get("druid.information." + nodeRole);
if (ann == null) throw new IllegalStateException("Missing druid info annotation for role " + nodeRole);
try { new ObjectMapper().readTree(ann); } catch (Exception e) { throw new IllegalStateException("Invalid JSON in annotation: " + e.getMessage()); }

boolean hasValidNodeAnnotation(Pod p, String role) {
  Map<String,String> anns = p.getMetadata() != null ? p.getMetadata().getAnnotations() : null;
  String json = anns == null ? null : anns.get("druid.information." + role);
  if (json == null) return false;
  try { JSON_MAPPER.readValue(json, DiscoveryDruidNode.class); return true; } catch (Exception e) { return false; }
}

try {
  DiscoveryDruidNode node = jsonMapper.readValue(jsonStr, DiscoveryDruidNode.class);
} catch (JsonProcessingException ex) {
  LOG.warn(ex, "Skipping pod with undecodable node annotation: %.100s", jsonStr);
  return null; // skip rather than fail the whole discovery iteration
}

When it happens

Trigger: Calling getDiscoveryDruidNodeFromPodDef for a pod whose info annotation contains invalid JSON, null (annotation absent), or a JSON payload that does not match DiscoveryDruidNode's schema (wrong fields/types). Raised from getDiscoveryDruidNodeFromPodDef, reached via node() and hasNext() while iterating discovered nodes.

Common situations: Mixed Druid cluster versions where an older node announces an annotation schema the reader can't parse; manual edits or truncated annotations; druid_k8s annotation keys changed between releases; pods from other frameworks accidentally carrying the label selector.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• Couldn't deserialize authorizer roleMap!
• Invalid JSON inside unknown key:
• unknown event type [%s]
• interrupted flushing elements from queue
• Cannot deserialize type[%s] to an RoaringBitmap64Counter:

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0d332bf681058843. Report an issue: GitHub.