apache/flink · error · IOException

Could not register schema in registry

Error message

Could not register schema in registry

What it means

IOException from ConfluentSchemaRegistryCoder.writeSchema when schemaRegistryClient.register(subject, schema) fails with RestClientException during message encoding. Registration is performed lazily on first record write, so DDL validation passes but the job fails at runtime.

Source

Thrown at flink-formats/flink-avro-confluent-registry/src/main/java/org/apache/flink/formats/avro/registry/confluent/ConfluentSchemaRegistryCoder.java:90

            try {
                return schemaRegistryClient.getById(schemaId);
            } catch (RestClientException e) {
                throw new IOException(
                        format("Could not find schema with id %s in registry", schemaId), e);
            }
        }
    }

    @Override
    public void writeSchema(Schema schema, OutputStream out) throws IOException {
        try {
            int registeredId = schemaRegistryClient.register(subject, schema);
            out.write(CONFLUENT_MAGIC_BYTE);
            byte[] schemaIdBytes = ByteBuffer.allocate(4).putInt(registeredId).array();
            out.write(schemaIdBytes);
        } catch (RestClientException e) {
            throw new IOException("Could not register schema in registry", e);
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Check the cause RestClientException for the HTTP status: 401/403 = auth or read-only registry, 409 = compatibility violation.
2. Pre-register the schema (e.g. with a Maven registry plugin or curl POST to /subjects/<subject>/versions) and disable auto-registration if the environment requires it.
3. Evolve schemas compatibly (add fields with defaults) or reset the subject's compatibility mode deliberately.
4. Verify registry URL, credentials, and network reachability from TaskManagers.

// pre-flight registration check before starting the job
try (CachedSchemaRegistryClient c = new CachedSchemaRegistryClient(url, 100)) {
    c.register(subject, schema); // fails fast at deploy time instead of first record
} catch (RestClientException e) {
    throw new IllegalStateException(
        "Cannot register schema under '" + subject + "': HTTP " + e.getStatus(), e);
}

try {
    serializationSchema.open(null);
    serializationSchema.serialize(row, out);
} catch (IOException e) {
    if (e.getMessage().equals("Could not register schema in registry")) {
        RestClientException rce = (RestClientException) e.getCause();
        // 401/403 auth or read-only; 409 compatibility conflict
        log.error("Registry registration failed: HTTP {}", rce.getStatus(), rce);
    }
    throw e;
}

When it happens

Trigger: First record emitted by an avro-confluent-registry sink when the registry is unreachable, credentials are rejected, or the schema is incompatible with the subject's configured compatibility mode; subjects in read-only mode on hosted registries.

Common situations: Schema evolution adding/removing fields under BACKWARD/FULL compatibility policy; wrong basic-auth/TLS settings; Confluent Cloud schema registry with disabled auto-registration.

Related errors

• Failed to serialize schema registry.
• Unknown data format. Magic number does not match
• Could not find schema with id %s in registry
• Option %s.%s is required for serialization
• Failed to deserialize Avro record.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/68ebe52de61762f5. Report an issue: GitHub.