grpc/grpc-java · error · ResourceInvalidException

Failed to parse Endpoint metadata: ${e.getMessage()}

Error message

Failed to parse Endpoint metadata: ${e.getMessage()}

What it means

When parsing each LbEndpoint of an EDS ClusterLoadAssignment, gRPC parses endpoint-level metadata via MetadataRegistry.parseMetadata. If the endpoint's metadata Struct is invalid, the entire EDS resource is rejected with this ResourceInvalidException and the update is NACKed.

Source

Thrown at xds/src/main/java/io/grpc/xds/XdsEndpointResource.java:224

    MetadataRegistry registry = MetadataRegistry.getInstance();
    try {
      localityMetadata = registry.parseMetadata(proto.getMetadata());
    } catch (ResourceInvalidException e) {
      throw new ResourceInvalidException("Failed to parse Locality Endpoint metadata: "
          + e.getMessage(), e);
    }
    List<Endpoints.LbEndpoint> endpoints = new ArrayList<>(proto.getLbEndpointsCount());
    for (io.envoyproxy.envoy.config.endpoint.v3.LbEndpoint endpoint : proto.getLbEndpointsList()) {
      // The endpoint field of each lb_endpoints must be set.
      // Inside of it: the address field must be set.
      if (!endpoint.hasEndpoint() || !endpoint.getEndpoint().hasAddress()) {
        return StructOrError.fromError("LbEndpoint with no endpoint/address");
      }
      ImmutableMap<String, Object> endpointMetadata;
      try {
        endpointMetadata = registry.parseMetadata(endpoint.getMetadata());
      } catch (ResourceInvalidException e) {
        throw new ResourceInvalidException("Failed to parse Endpoint metadata: "
            + e.getMessage(), e);
      }
      List<java.net.SocketAddress> addresses = new ArrayList<>();
      addresses.add(getInetSocketAddress(endpoint.getEndpoint().getAddress()));

      if (isEnabledXdsDualStack()) {
        for (Endpoint.AdditionalAddress additionalAddress
            : endpoint.getEndpoint().getAdditionalAddressesList()) {
          addresses.add(getInetSocketAddress(additionalAddress.getAddress()));
        }
      }
      boolean isHealthy = (endpoint.getHealthStatus() == HealthStatus.HEALTHY)
              || (endpoint.getHealthStatus() == HealthStatus.UNKNOWN);
      endpoints.add(Endpoints.LbEndpoint.create(
          new EquivalentAddressGroup(addresses),
          endpoint.getLoadBalancingWeight().getValue(), isHealthy,
          endpoint.getEndpoint().getHostname(),
          endpointMetadata));

View on GitHub (pinned to 64daddc1f3)

Solutions

1. Read the appended cause message to identify the exact metadata key/value that failed
2. Correct the per-endpoint metadata Struct in the EDS resource on the management server
3. Remove unsupported metadata keys or encode values with supported Struct types
4. Re-send the corrected resource so gRPC ACKs the update

Example fix

// before
lb_endpoints { metadata { fields { key: "weight" value { string_value: "ten" } } } }
// after
lb_endpoints { metadata { fields { key: "weight" value { number_value: 10 } } } }

// Control-plane side: validate endpoint metadata before sending
MetadataRegistry registry = MetadataRegistry.getInstance();
for (var lbEndpoint : loadAssignment.getEndpoints(0).getLbEndpointsList()) {
  try {
    registry.parseMetadata(lbEndpoint.getMetadata());
  } catch (ResourceInvalidException e) {
    throw new IllegalStateException("Invalid endpoint metadata: " + e.getMessage(), e);
  }
}

boolean isValidEndpointMetadata(com.google.protobuf.Struct metadata) {
  try {
    MetadataRegistry.getInstance().parseMetadata(metadata);
    return true;
  } catch (ResourceInvalidException e) {
    return false;
  }
}

// Client side: capture the NACK reason from the watcher error status
@Override public void onError(Status error) {
  if (error.getDescription().contains("Failed to parse Endpoint metadata")) {
    logger.log(WARNING, "Bad endpoint metadata in EDS resource: " + error.getDescription());
  }
}

When it happens

Trigger: An EDS resource arrives where lb_endpoints[i].metadata contains a Struct that fails MetadataRegistry.parseMetadata (unsupported value type, invalid nested structure, or a field violating the metadata schema). The exception is rethrown with this prefix in parseLocalityLbEndpoints.

Common situations: Custom xDS management server attaches endpoint metadata in a non-conforming format; Istio/Envoy emits metadata extensions gRPC does not accept; manually crafted load assignment protos with garbage metadata fields.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Failed to parse Locality Endpoint metadata: ${e.getMessage()
• Failed to parse metadata key: %s, type: %s. Error: %s
• Failed to parse xDS filter metadata for cluster '" + cluster
• Invalid message type: ${unpackedMessage.getClass()}
• Address is not an IP

AI-assisted analysis of grpc/grpc-java@64daddc1f3 (2026-09-08). Data as JSON: /api/errors/3fb33f6dff3b4c04. Report an issue: GitHub.