apache/pulsar · error · IncompatibleSchemaException

Incompatible schema: existing schema is not in valid Avro fo

Error message

Incompatible schema: existing schema is not in valid Avro format for SchemaType.JSON

What it means

Defense-in-depth check (PIP-464): when the legacy Jackson JSON schema format is disabled on the broker, an already-stored JSON schema that is not valid Avro cannot be compared at all, so the update is rejected rather than silently accepted.

Source

Thrown at pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/JsonSchemaCompatibilityCheck.java:83

                throw new IncompatibleSchemaException(
                        "Incompatible schema: expected Avro schema format for SchemaType.JSON");
            }
        } else if (allowLegacyJacksonFormat && isJsonSchema(from)) {

            if (isAvroSchema(to)) {
                // if broker have the schema in old json format but producer sent a schema in the avro format
                // return true and overwrite the old format
            } else if (isJsonSchema(to)) {
                // if both producer and broker have the schema in old json format
                isCompatibleJsonSchema(from, to);
            } else {
                throw new IncompatibleSchemaException(
                        "Incompatible schema: expected Avro schema format for SchemaType.JSON");
            }
        } else if (!allowLegacyJacksonFormat && !isAvroSchema(from)) {
            // When legacy format is disabled, the existing schema must be valid Avro.
            // If it's not, this is a defense-in-depth rejection (PIP-464).
            throw new IncompatibleSchemaException(
                    "Incompatible schema: existing schema is not in valid Avro format for SchemaType.JSON");
        } else {
            // broker has schema format with unknown format
            // maybe corrupted?
            // return true to overwrite
        }
    }

    private static final ObjectReader JSON_SCHEMA_READER =
            ObjectMapperFactory.getMapper().reader().forType(JsonSchema.class);
    private void isCompatibleJsonSchema(SchemaData from, SchemaData to) throws IncompatibleSchemaException {
        try {
            JsonSchema fromSchema = JSON_SCHEMA_READER.readValue(from.getData());
            JsonSchema toSchema = JSON_SCHEMA_READER.readValue(to.getData());
            if (!fromSchema.getId().equals(toSchema.getId())) {
                throw new IncompatibleSchemaException(String.format("Incompatible Schema from %s + to %s",
                        new String(from.getData(), UTF_8), new String(to.getData(), UTF_8)));
            }

View on GitHub (pinned to 820761864e)

Solutions

1. Re-register the topic schema in Avro format: delete the schema and upload a modern Avro-encoded JSON schema.
2. Temporarily enable allowLegacyJacksonFormat on the broker so the legacy schema can be overwritten by an Avro one.
3. Migrate affected topics to fresh schemas created with a current client before disabling legacy format.

Example fix

// before: legacy schema left in place, legacy format disabled
// after: re-upload in Avro format
SchemaInfo info = Schema.JSON(MyPojo.class).getSchemaInfo();
admin.schemas().createSchema("persistent://tenant/ns/topic", info);

SchemaInfo current = admin.schemas().getSchemaInfo(topic);
String def = new String(current.getSchema(), StandardCharsets.UTF_8);
boolean validAvro;
try { new Schema.Parser().parse(def); validAvro = true; } catch (Exception e) { validAvro = false; }
if (!validAvro) { /* re-register schema in Avro format before further updates */ }

boolean storedSchemaIsAvroJson(SchemaInfo info) {
    if (info.getType() != SchemaType.JSON) return false;
    try { new org.apache.avro.Schema.Parser().parse(new String(info.getSchema(), java.nio.charset.StandardCharsets.UTF_8)); return true; }
    catch (Exception e) { return false; }
}

try {
    admin.schemas().createSchema(topic, newInfo);
} catch (PulsarAdminException e) {
    if (e.getMessage().contains("existing schema is not in valid Avro format")) {
        admin.schemas().deleteSchema(topic); // then re-create in Avro format
        admin.schemas().createSchema(topic, Schema.JSON(Pojo.class).getSchemaInfo());
    }
}

When it happens

Trigger: checkCompatible where allowLegacyJacksonFormat is false and isAvroSchema(from) is false — i.e. the existing stored schema for the topic is a legacy Jackson-format (or corrupt) JSON schema and any new schema update arrives.

Common situations: Broker upgraded to a version that disables the legacy format while old topics still carry Jackson-format JSON schemas; schemas created by very old Pulsar clients; corrupted schema metadata after restore/migration.

Related errors

• Incompatible schema: expected Avro schema format for SchemaT
• Incompatible Schema from %s + to %s
• Incompatible schema: exists schema type %s, new schema type
• Avro schema typed [UNION] is not supported
• Avro Record Builder doesn't support non-avro record as a fie

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/5c60b3488e69900d. Report an issue: GitHub.