apache/flink · error · IOException

Could not find schema with id %s in registry

Error message

Could not find schema with id %s in registry

What it means

IOException from ConfluentSchemaRegistryCoder.readSchema when schemaRegistryClient.getById(schemaId) throws RestClientException: the id encoded in the message cannot be resolved to a schema by the configured Schema Registry. Usually the registry is wrong/unreachable or the schema id was deleted/never existed there.

Source

Thrown at flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/ConfluentSchemaRegistryCoder.java:76

     * @param schemaRegistryClient client to connect schema registry
     */
    public ConfluentSchemaRegistryCoder(SchemaRegistryClient schemaRegistryClient) {
        this.schemaRegistryClient = schemaRegistryClient;
    }

    @Override
    public Schema readSchema(InputStream in) throws IOException {
        DataInputStream dataInputStream = new DataInputStream(in);

        if (dataInputStream.readByte() != 0) {
            throw new IOException("Unknown data format. Magic number does not match");
        } else {
            int schemaId = dataInputStream.readInt();

            try {
                return schemaRegistryClient.getById(schemaId);
            } catch (RestClientException e) {
                throw new IOException(
                        format("Could not find schema with id %s in registry", schemaId), e);
            }
        }
    }

    @Override
    public void writeSchema(Schema schema, OutputStream out) throws IOException {
        try {
            int registeredId = schemaRegistryClient.register(subject, schema);
            out.write(CONFLUENT_MAGIC_BYTE);
            byte[] schemaIdBytes = ByteBuffer.allocate(4).putInt(registeredId).array();
            out.write(schemaIdBytes);
        } catch (RestClientException e) {
            throw new IOException("Could not register schema in registry", e);
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Verify the schema id exists: curl -u user:pass <registry-url>/schemas/ids/<id>.
2. Confirm 'avro-confluent-registry.schema-registry.url' (plus optional basic-auth credentials) points to the same registry the producer registered against.
3. If the schema was deleted, restore/re-register it under the original id or re-produce the data.
4. For transient registry outages, restart the job — Flink will retry from the last checkpoint.

// pre-flight: can the configured registry resolve the schema ids you expect?
try (CachedSchemaRegistryClient c = new CachedSchemaRegistryClient(url, 100)) {
    Schema s = c.getById(expectedSchemaId); // throws if unknown
} catch (RestClientException e) {
    throw new IllegalStateException("Schema id " + expectedSchemaId + " not resolvable", e);
}

try {
    ... // deserialization loop
} catch (IOException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("Could not find schema")) {
        // registry/env mismatch: check URL, auth, and id existence; do not skip the record
        throw new IllegalStateException("Fatal: schema id unknown to registry", e);
    }
    throw e;
}

When it happens

Trigger: Reading Confluent-wire-format messages whose schema id is unknown to the configured registry URL; pointing at a different registry environment (prod vs test); schema deleted by registry cleanup; transient registry HTTP errors surface as the same RestClientException.

Common situations: Environment mismatch between producing and consuming clusters; registry URL typo; auth (basic auth/TLS) not configured so registry returns 401/403; soft-deleted schemas after subject deletion.

Related errors

• Failed to serialize schema registry.
• Unknown data format. Magic number does not match
• Could not register schema in registry
• Option %s.%s is required for serialization
• Failed to deserialize Avro record.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/2e9380820e4e016a. Report an issue: GitHub.