apache/beam · error · RuntimeException

Unable to get latest schema metadata for subject:

Error message

Unable to get latest schema metadata for subject: 

What it means

ConfluentSchemaRegistryDeserializerProvider.getSchemaMetadata() fetches the schema metadata from the Confluent Schema Registry, either the latest version or a pinned version. Any IOException or RestClientException from the registry client is rethrown as a RuntimeException prefixed "Unable to get latest schema metadata for subject: ". This indicates the pipeline could not resolve the subject's schema at startup.

Solutions

1. Verify the subject exists: curl the registry at /subjects and /subjects/<subject>/versions.
2. Check the registry URL, credentials (auth), and TLS trust configuration passed to the provider.
3. If a specific version was pinned, confirm that version still exists (registry cleanup may have removed it) or drop the version pin to use latest.
4. Confirm workers can reach the registry host/port (network/VPC/firewall).

Example fix

// before
ConfluentSchemaRegistryDeserializerProvider.of("bad-url", "my-topic-value", null, ...)
// after
ConfluentSchemaRegistryDeserializerProvider.of("https://schema-registry:8081", "my-topic-value", null,
  Collections.singletonMap("basic.auth.credentials.source", "USER_INFO"), ...)

// pre-flight check against the registry
HttpGet req = new HttpGet(registryUrl + "/subjects/" + subject + "/versions/latest");
// expect 200 with auth headers; 404 means subject missing

try { getAvroSchema(subject); }
catch (RuntimeException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Unable to get latest schema metadata")) {
    // check subject existence / registry reachability, retry with backoff
  }
  throw e;
}

When it happens

Trigger: getSchemaMetadata() called (directly or via getAvroSchema) when the schema registry is unreachable, the subject does not exist, the version doesn't exist, or auth/TLS to the registry fails.

Common situations: Wrong schema.registry.url; subject renamed or deleted (schema cleanup policies); missing basic auth credentials; registry behind TLS with untrusted certs; network segmentation between workers and registry.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• Exception while reading from Kafka
• Failed to download file
• : Timeout while initializing partition ' '. Kafka client…
• Artifact not found at
• Can not get unique key from solr

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a3c516053bc6523b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/ConfluentSchemaRegistryDeserializerProvider.java:147

    return deserializer;
  }

  @Override
  public Coder<T> getCoder(CoderRegistry coderRegistry) {
    return (Coder<T>) AvroCoder.of(getAvroSchema());
  }

  private Schema getAvroSchema() {
    return new Schema.Parser().parse(getSchemaMetadata().getSchema());
  }

  private SchemaMetadata getSchemaMetadata() {
    try {
      return (version == null)
          ? getSchemaRegistryClient().getLatestSchemaMetadata(subject)
          : getSchemaRegistryClient().getSchemaMetadata(subject, version);
    } catch (IOException | RestClientException e) {
      throw new RuntimeException("Unable to get latest schema metadata for subject: " + subject, e);
    }
  }

  private SchemaRegistryClient getSchemaRegistryClient() {
    return this.schemaRegistryClientProviderFn.apply(null);
  }
}

class ConfluentSchemaRegistryDeserializer extends KafkaAvroDeserializer {
  Schema readerSchema;

  ConfluentSchemaRegistryDeserializer(SchemaRegistryClient client, Schema readerSchema) {
    super(client);
    this.readerSchema = readerSchema;
  }

  @Override
  public Object deserialize(String s, byte[] bytes) {

View on GitHub (pinned to 12126d8942)