apache/druid · error · ParseException

Fail to get protobuf schema because of can not connect to…

Error message

Fail to get protobuf schema because of can not connect to registry or failed http request!

What it means

parse() fetches the protobuf schema for the id embedded in the Confluent wire-format envelope via registry.getSchemaById(id). If the schema registry REST call fails (RestClientException — connectivity problems, unknown id, bad HTTP response), the decoder wraps it in a ParseException saying it could not get the protobuf schema.

Solutions

Verify 'urls' points to the schema registry that actually produced the messages and that it is reachable from Druid workers (curl the /schemas/ids/<id> endpoint)
Re-register the missing schema / restore the subject if it was deleted
Check registry auth (credentials/basic auth config) and network policies; increase timeouts if the registry is slow

Example fix

// before: wrong environment registry
"schemaRegistry": {"urls": ["http://staging-registry:8081"]}
// after
"schemaRegistry": {"urls": ["http://prod-registry:8081"], "config": {"basic.auth.credentials.source": "USER_INFO", "basic.auth.user.info": "user:pass"}}

Defensive patterns

Strategy: try-catch

Validate before calling

int schemaId = ByteBuffer.wrap(bytes, 1, 4).getInt();
curl -f http://registry:8081/schemas/ids/<schemaId>  // must return 200 with the schema

Try / catch

try {
  byte[] parsed = decoder.parse(bytes);
} catch (ParseException e) {
  if (e.getMessage().contains("can not connect to registry")) {
    log.error("Schema registry unreachable or schema id missing; check urls/network", e.getCause());
    // retry with backoff or route to DLQ
  }
}

Prevention

Pre-fetch every schema id expected on the topic via the registry REST API before starting ingestion
Open network/firewall access from Druid workers to the registry and set sane timeouts
Disable subject/schema deletion (or use immortal cleanup policy) for production subjects

When it happens

Trigger: Consuming a message whose 4-byte schema id is not present in the configured registry (wrong registry, schema deleted, subject purged) or the registry is unreachable/timing out, raising RestClientException inside SchemaRegistryBasedProtobufBytesDecoder.parse.

Common situations: Pointing Druid at the wrong schema registry environment; schema auto-deregistered due to cleanup policies; network/firewall blocking Druid workers from the registry; registry returning 40403 unknown schema for an id.

Understand the failure class

Background: 'Something went wrong' / 'Request failed (500)' / 'HTTP error! status: 404' — what failed HTTP requests actually mean and how to find the real cause — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/649f853652ee2978. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/protobuf-extensions/src/main/java/org/apache/druid/data/input/protobuf/SchemaRegistryBasedProtobufBytesDecoder.java:187

   * @param bytes ByteBuffer containing the Confluent Schema Registry formatted protobuf message
   * @return DynamicMessage parsed from the protobuf bytes
   * @throws ParseException if the schema cannot be retrieved or the message cannot be parsed
   */
  @Override
  public DynamicMessage parse(ByteBuffer bytes)
  {
    bytes.get(); // ignore first \0 byte
    int id = bytes.getInt(); // extract schema registry id
    bytes.get(); // ignore \0 byte before PB message
    int length = bytes.limit() - 2 - 4;
    Descriptors.Descriptor descriptor;
    try {
      ProtobufSchema schema = (ProtobufSchema) registry.getSchemaById(id);
      descriptor = schema.toDescriptor();
    }
    catch (RestClientException e) {
      LOGGER.error(e.getMessage());
      throw new ParseException(
          null,
          e,
          "Fail to get protobuf schema because of can not connect to registry or failed http request!"
      );
    }
    catch (IOException e) {
      LOGGER.error(e.getMessage());
      throw new ParseException(null, e, "Fail to get protobuf schema because of invalid schema!");
    }
    try {
      byte[] rawMessage = new byte[length];
      bytes.get(rawMessage, 0, length);
      return DynamicMessage.parseFrom(descriptor, rawMessage);
    }
    catch (Exception e) {
      LOGGER.error(e.getMessage());
      throw new ParseException(null, e, "Fail to decode protobuf message!");
    }

View on GitHub (pinned to 9b90983fd2)