apache/druid · warning

KV entry [%s] not found for service metadata

Error message

KV entry [%s] not found for service metadata

What it means

While translating Consul health-check results into Druid service instances, DefaultConsulApiClient.parseHealthServices looks up a per-service metadata KV entry (kvKey) expected to hold the node's JSON metadata (base64-encoded). If the KV read returns no value, this warning is logged and that service instance is skipped (continue) rather than failing the whole query. It indicates Consul health says a node is healthy, but its discovery metadata entry is missing from the KV store.

Source

Thrown at extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java:264

            healthService.getService().getMeta() == null) {
          continue;
        }

        Map<String, String> meta = healthService.getService().getMeta();
        final String nodeJson;

        if (meta.containsKey("druid_node")) {
          nodeJson = meta.get("druid_node");
        } else if (meta.containsKey("druid_node_kv")) {
          String kvKey = meta.get("druid_node_kv");
          Response<GetValue> kvResponse = consulClient.getKVValue(kvKey, config.getAuth().getAclToken(), buildQueryParams());
          if (kvResponse != null && kvResponse.getValue() != null && kvResponse.getValue().getValue() != null) {
            nodeJson = new String(
                Base64.getDecoder().decode(kvResponse.getValue().getValue()),
                StandardCharsets.UTF_8
            );
          } else {
            LOGGER.warn("KV entry [%s] not found for service metadata", kvKey);
            continue;
          }
        } else {
          continue;
        }

        DiscoveryDruidNode node = jsonMapper.readValue(nodeJson, DiscoveryDruidNode.class);
        nodes.add(node);
      }
      catch (IOException e) {
        LOGGER.error(e, "Failed to parse DiscoveryDruidNode from Consul service metadata");
      }
      catch (Exception e) {
        LOGGER.error(e, "Failed to retrieve or parse DiscoveryDruidNode from Consul");
      }
    }

    return nodes;

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Verify the expected KV key exists: consul kv get <kvKey>; if missing, restart or re-register the affected Druid node so it rewrites its metadata
2. Check the Consul ACL token — reads denied by ACL can surface as null values; grant read on the discovery KV prefix
3. Confirm the KV path convention matches between writer (node registration) and reader (parseHealthServices); fix config if paths diverge after an upgrade or namespace change
4. Check Consul UI/health endpoint to see whether the affected node is still alive; if it's a zombie registration, deregister it (consul services deregister <id>)

Example fix

// before: node registered but metadata key absent
$ consul catalog services   # druid/historical listed
$ consul kv get druid/discovery/druid/historical/node1   # (empty)
// after: restart the node so it rewrites metadata
$ consul kv get druid/discovery/druid/historical/node1   # {"host":"...","port":...}

// before relying on healthy-services lookups, verify every healthy instance has metadata:
for (String node : healthyServiceIds) {
  String kvKey = "druid/discovery/" + serviceName + "/" + node;
  Response<GetValue> kv = consulClient.getKVValue(kvKey, aclToken, qp);
  if (kv == null || kv.getValue() == null || kv.getValue().getValue() == null) {
    throw new IllegalStateException("Missing discovery metadata KV: " + kvKey);
  }
}

static boolean hasKvValue(Response<GetValue> r) {
  return r != null && r.getValue() != null && r.getValue().getValue() != null;
}

// the client already warns-and-skips; wrap higher-level discovery calls if a full result set is required:
List<ServiceEntity> services = discoveryClient.getHealthyServices(serviceName);
if (services.isEmpty() && expectedMinNodes > 0) {
  throw new IllegalStateException("All services skipped due to missing KV metadata");
}

When it happens

Trigger: getHealthyServices or nodes lists healthy Consul services, and for one of them the companion KV lookup (e.g. druid/discovery/<service>/<node>) returns a response whose getValue()/getValue().getValue() is null — the metadata key was never written, was deleted, or the ACL token cannot read that KV prefix.

Common situations: A Druid node registered with Consul but crashed before writing its metadata KV; metadata keys cleaned up by a TTL job or manual purge; Consul ACL token lacking read permission on the discovery KV prefix so reads silently return no value; version/config mismatch where services register under a different KV path than the reader queries.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• can't stop.
• can't start
• can't stop
• FilteringUpstreamListener should not be registered with Serv
• Cannot start once already started or closed

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8b771c0ba3d8e05e. Report an issue: GitHub.