{"record":{"id":"8b771c0ba3d8e05e","repo":"apache/druid","slug":"kv-entry-s-not-found-for-service-metadata","errorCode":null,"errorMessage":"KV entry [%s] not found for service metadata","messagePattern":"KV entry \\[(.+?)\\] not found for service metadata","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java","lineNumber":264,"sourceCode":"            healthService.getService().getMeta() == null) {\n          continue;\n        }\n\n        Map<String, String> meta = healthService.getService().getMeta();\n        final String nodeJson;\n\n        if (meta.containsKey(\"druid_node\")) {\n          nodeJson = meta.get(\"druid_node\");\n        } else if (meta.containsKey(\"druid_node_kv\")) {\n          String kvKey = meta.get(\"druid_node_kv\");\n          Response<GetValue> kvResponse = consulClient.getKVValue(kvKey, config.getAuth().getAclToken(), buildQueryParams());\n          if (kvResponse != null && kvResponse.getValue() != null && kvResponse.getValue().getValue() != null) {\n            nodeJson = new String(\n                Base64.getDecoder().decode(kvResponse.getValue().getValue()),\n                StandardCharsets.UTF_8\n            );\n          } else {\n            LOGGER.warn(\"KV entry [%s] not found for service metadata\", kvKey);\n            continue;\n          }\n        } else {\n          continue;\n        }\n\n        DiscoveryDruidNode node = jsonMapper.readValue(nodeJson, DiscoveryDruidNode.class);\n        nodes.add(node);\n      }\n      catch (IOException e) {\n        LOGGER.error(e, \"Failed to parse DiscoveryDruidNode from Consul service metadata\");\n      }\n      catch (Exception e) {\n        LOGGER.error(e, \"Failed to retrieve or parse DiscoveryDruidNode from Consul\");\n      }\n    }\n\n    return nodes;","sourceCodeStart":246,"sourceCodeEnd":282,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-contrib/consul-extensions/src/main/java/org/apache/druid/consul/discovery/DefaultConsulApiClient.java#L246-L282","documentation":"While translating Consul health-check results into Druid service instances, DefaultConsulApiClient.parseHealthServices looks up a per-service metadata KV entry (kvKey) expected to hold the node's JSON metadata (base64-encoded). If the KV read returns no value, this warning is logged and that service instance is skipped (continue) rather than failing the whole query. It indicates Consul health says a node is healthy, but its discovery metadata entry is missing from the KV store.","triggerScenarios":"getHealthyServices or nodes lists healthy Consul services, and for one of them the companion KV lookup (e.g. druid/discovery/<service>/<node>) returns a response whose getValue()/getValue().getValue() is null — the metadata key was never written, was deleted, or the ACL token cannot read that KV prefix.","commonSituations":"A Druid node registered with Consul but crashed before writing its metadata KV; metadata keys cleaned up by a TTL job or manual purge; Consul ACL token lacking read permission on the discovery KV prefix so reads silently return no value; version/config mismatch where services register under a different KV path than the reader queries.","solutions":["Verify the expected KV key exists: consul kv get <kvKey>; if missing, restart or re-register the affected Druid node so it rewrites its metadata","Check the Consul ACL token — reads denied by ACL can surface as null values; grant read on the discovery KV prefix","Confirm the KV path convention matches between writer (node registration) and reader (parseHealthServices); fix config if paths diverge after an upgrade or namespace change","Check Consul UI/health endpoint to see whether the affected node is still alive; if it's a zombie registration, deregister it (consul services deregister <id>)"],"exampleFix":"// before: node registered but metadata key absent\n$ consul catalog services   # druid/historical listed\n$ consul kv get druid/discovery/druid/historical/node1   # (empty)\n// after: restart the node so it rewrites metadata\n$ consul kv get druid/discovery/druid/historical/node1   # {\"host\":\"...\",\"port\":...}","handlingStrategy":"validation","validationCode":"// before relying on healthy-services lookups, verify every healthy instance has metadata:\nfor (String node : healthyServiceIds) {\n  String kvKey = \"druid/discovery/\" + serviceName + \"/\" + node;\n  Response<GetValue> kv = consulClient.getKVValue(kvKey, aclToken, qp);\n  if (kv == null || kv.getValue() == null || kv.getValue().getValue() == null) {\n    throw new IllegalStateException(\"Missing discovery metadata KV: \" + kvKey);\n  }\n}","typeGuard":"static boolean hasKvValue(Response<GetValue> r) {\n  return r != null && r.getValue() != null && r.getValue().getValue() != null;\n}","tryCatchPattern":"// the client already warns-and-skips; wrap higher-level discovery calls if a full result set is required:\nList<ServiceEntity> services = discoveryClient.getHealthyServices(serviceName);\nif (services.isEmpty() && expectedMinNodes > 0) {\n  throw new IllegalStateException(\"All services skipped due to missing KV metadata\");\n}","preventionTips":["Verify the ACL token has read access to the discovery KV prefix","Ensure nodes write their metadata KV before/atomically with Consul registration","Confirm writer and reader agree on the KV path convention, especially after upgrades or namespace changes","Deregister stale/zombie service instances from the Consul catalog"],"tags":["consul","service-discovery","kv-store","missing-metadata"],"backgroundTag":"empty-result-set","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}