{"record":{"id":"5c60b3488e69900d","repo":"apache/pulsar","slug":"incompatible-schema-existing-schema-is-not-in-val","errorCode":null,"errorMessage":"Incompatible schema: existing schema is not in valid Avro format for SchemaType.JSON","messagePattern":"Incompatible schema: existing schema is not in valid Avro format for SchemaType\\.JSON","errorType":"exception","errorClass":"IncompatibleSchemaException","httpStatus":null,"severity":"error","filePath":"pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/JsonSchemaCompatibilityCheck.java","lineNumber":83,"sourceCode":"                throw new IncompatibleSchemaException(\n                        \"Incompatible schema: expected Avro schema format for SchemaType.JSON\");\n            }\n        } else if (allowLegacyJacksonFormat && isJsonSchema(from)) {\n\n            if (isAvroSchema(to)) {\n                // if broker have the schema in old json format but producer sent a schema in the avro format\n                // return true and overwrite the old format\n            } else if (isJsonSchema(to)) {\n                // if both producer and broker have the schema in old json format\n                isCompatibleJsonSchema(from, to);\n            } else {\n                throw new IncompatibleSchemaException(\n                        \"Incompatible schema: expected Avro schema format for SchemaType.JSON\");\n            }\n        } else if (!allowLegacyJacksonFormat && !isAvroSchema(from)) {\n            // When legacy format is disabled, the existing schema must be valid Avro.\n            // If it's not, this is a defense-in-depth rejection (PIP-464).\n            throw new IncompatibleSchemaException(\n                    \"Incompatible schema: existing schema is not in valid Avro format for SchemaType.JSON\");\n        } else {\n            // broker has schema format with unknown format\n            // maybe corrupted?\n            // return true to overwrite\n        }\n    }\n\n    private static final ObjectReader JSON_SCHEMA_READER =\n            ObjectMapperFactory.getMapper().reader().forType(JsonSchema.class);\n    private void isCompatibleJsonSchema(SchemaData from, SchemaData to) throws IncompatibleSchemaException {\n        try {\n            JsonSchema fromSchema = JSON_SCHEMA_READER.readValue(from.getData());\n            JsonSchema toSchema = JSON_SCHEMA_READER.readValue(to.getData());\n            if (!fromSchema.getId().equals(toSchema.getId())) {\n                throw new IncompatibleSchemaException(String.format(\"Incompatible Schema from %s + to %s\",\n                        new String(from.getData(), UTF_8), new String(to.getData(), UTF_8)));\n            }","sourceCodeStart":65,"sourceCodeEnd":101,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/schema/JsonSchemaCompatibilityCheck.java#L65-L101","documentation":"Defense-in-depth check (PIP-464): when the legacy Jackson JSON schema format is disabled on the broker, an already-stored JSON schema that is not valid Avro cannot be compared at all, so the update is rejected rather than silently accepted.","triggerScenarios":"checkCompatible where allowLegacyJacksonFormat is false and isAvroSchema(from) is false — i.e. the existing stored schema for the topic is a legacy Jackson-format (or corrupt) JSON schema and any new schema update arrives.","commonSituations":"Broker upgraded to a version that disables the legacy format while old topics still carry Jackson-format JSON schemas; schemas created by very old Pulsar clients; corrupted schema metadata after restore/migration.","solutions":["Re-register the topic schema in Avro format: delete the schema and upload a modern Avro-encoded JSON schema.","Temporarily enable allowLegacyJacksonFormat on the broker so the legacy schema can be overwritten by an Avro one.","Migrate affected topics to fresh schemas created with a current client before disabling legacy format."],"exampleFix":"// before: legacy schema left in place, legacy format disabled\n// after: re-upload in Avro format\nSchemaInfo info = Schema.JSON(MyPojo.class).getSchemaInfo();\nadmin.schemas().createSchema(\"persistent://tenant/ns/topic\", info);","handlingStrategy":"validation","validationCode":"SchemaInfo current = admin.schemas().getSchemaInfo(topic);\nString def = new String(current.getSchema(), StandardCharsets.UTF_8);\nboolean validAvro;\ntry { new Schema.Parser().parse(def); validAvro = true; } catch (Exception e) { validAvro = false; }\nif (!validAvro) { /* re-register schema in Avro format before further updates */ }","typeGuard":"boolean storedSchemaIsAvroJson(SchemaInfo info) {\n    if (info.getType() != SchemaType.JSON) return false;\n    try { new org.apache.avro.Schema.Parser().parse(new String(info.getSchema(), java.nio.charset.StandardCharsets.UTF_8)); return true; }\n    catch (Exception e) { return false; }\n}","tryCatchPattern":"try {\n    admin.schemas().createSchema(topic, newInfo);\n} catch (PulsarAdminException e) {\n    if (e.getMessage().contains(\"existing schema is not in valid Avro format\")) {\n        admin.schemas().deleteSchema(topic); // then re-create in Avro format\n        admin.schemas().createSchema(topic, Schema.JSON(Pojo.class).getSchemaInfo());\n    }\n}","preventionTips":["Audit topics for legacy Jackson-format JSON schemas before upgrading brokers/PIP-464 changes.","Re-create old topic schemas in Avro format proactively.","Enable allowLegacyJacksonFormat during migration, then disable after cleanup.","Monitor schema upload failures after broker upgrades."],"tags":["pulsar","json-schema","avro","pip-464"],"backgroundTag":"json-schema-format-mismatch","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}