apache/druid · error · ParseException

Failed to fetch Avro schema id[%s] from registry. Check if t

Error message

Failed to fetch Avro schema id[%s] from registry. Check if the schema exists in the registry. Otherwise it could mean that there is malformed data in the stream or data that doesn't conform to the schema specified.

What it means

Thrown when IOException occurs while calling registry.getSchemaById(id) to fetch the Avro schema from the Confluent schema registry. Usually means the schema id does not exist in the registry, the registry is unreachable, or the stream contains malformed/nonconforming data.

Source

Thrown at extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/SchemaRegistryBasedAvroBytesDecoder.java:143

  @Override
  public GenericRecord parse(ByteBuffer bytes)
  {
    int length = bytes.limit() - 1 - 4;
    if (length < 0) {
      throw new ParseException(null, "Failed to decode avro message, not enough bytes to decode (%s)", bytes.limit());
    }

    bytes.get(); // ignore first \0 byte
    int id = bytes.getInt(); // extract schema registry id
    int offset = bytes.position() + bytes.arrayOffset();
    Schema schema;

    try {
      ParsedSchema parsedSchema = registry.getSchemaById(id);
      schema = parsedSchema instanceof AvroSchema ? ((AvroSchema) parsedSchema).rawSchema() : null;
    }
    catch (IOException ex1) {
      throw new ParseException(
          null,
          ex1,
          "Failed to fetch Avro schema id[%s] from registry. Check if the schema exists in the registry. Otherwise it"
          + " could mean that there is malformed data in the stream or data that doesn't conform to the schema"
          + " specified.",
          id
      );
    }
    catch (RestClientException ex2) {
      if (ex2.getErrorCode() == 401) {
        throw new ParseException(
            null,
            ex2,
            "Failed to authenticate to schema registry for Avro schema id[%s]. Please check your credentials.",
            id
        );
      }
      // For all other errors, just include the code and message received from the library.

View on GitHub (pinned to 9b90983fd2)

Solutions

Verify the schema id exists: curl http://<registry>/schemas/ids/<id>
Check the schema-registry URL and credentials in the AvroBytesDecoder inputFormat config
Confirm network connectivity from Druid processes to the schema registry host/port
Re-register missing schemas or repartition data produced with deleted schema ids

Example fix

// before
"avroBytesDecoder": {"type": "schema_registry", "schemaRegistryUrls": ["http://wrong-registry:8081"]}
// after
"avroBytesDecoder": {"type": "schema_registry", "schemaRegistryUrls": ["http://schema-registry:8081"]}

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight: check registry reachability and schema existence
curl -f http://schema-registry:8081/schemas/ids/1 > /dev/null && echo OK || echo FAIL

Try / catch

try {
  GenericRecord record = decoder.parse(bytes);
} catch (ParseException e) {
  if (e.getMessage().startsWith("Failed to fetch Avro schema")) {
    LOG.error(e, "Schema id missing/unreachable registry; pausing ingestion for ops check");
  }
}

Prevention

Add schema-registry health checks to deployment readiness probes
Use the same registry instance as producers
Alert on 404 schema fetches to catch deleted schemas
Avoid rebuilding registries without exporting/importing existing schemas

When it happens

Trigger: parse(ByteBuffer bytes) extracts a schema id from the wire-format header, then registry.getSchemaById(id) throws IOException — id not present in the target registry, wrong registry URL configured, or network failure.

Common situations: Pointing Druid at the wrong schema registry (different cluster/env); schema was deleted from the registry; topic data produced against a registry that was later rebuilt; DNS/network issues from Druid to the registry.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/6547a078679694ca. Report an issue: GitHub.