apache/pulsar · error · RuntimeException

Cannot convert ${m} to AVRO ${e.getMessage()}

Error message

Cannot convert ${m} to AVRO ${e.getMessage()}

What it means

jsonToAvro converts each JSON message body into AVRO-encoded bytes using an Avro writer; any IOException during encoding/flushing is rethrown as a RuntimeException 'Cannot convert <message> to AVRO <reason>'. It means one of the supplied message payloads could not be encoded against the configured AVRO schema.

Source

Thrown at pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/CmdProduce.java:231

        try {
            GenericDatumReader<Object> reader = new GenericDatumReader<>(avroSchema);
            JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(avroSchema, m);
            GenericDatumWriter<Object> writer = new GenericDatumWriter<>(avroSchema);
            ByteArrayOutputStream out = new ByteArrayOutputStream();
            Encoder e = EncoderFactory.get().binaryEncoder(out, null);
            Object datum = null;
            while (true) {
                try {
                    datum = reader.read(datum, jsonDecoder);
                } catch (EOFException eofException) {
                    break;
                }
                writer.write(datum, e);
                e.flush();
            }
            return out.toByteArray();
        } catch (IOException e) {
            throw new RuntimeException("Cannot convert " + m + " to AVRO " + e.getMessage(), e);
        }
    }

    @Spec
    private CommandSpec commandSpec;

    /**
     * Run the producer.
     *
     * @return 0 for success, < 0 otherwise
     * @throws Exception
     */
    @SuppressWarnings({"rawtypes", "unchecked"})
    public int run() throws PulsarClientException {
        if (this.numTimesProduce <= 0) {
            throw new CommandLine.ParameterException(commandSpec.commandLine(),
                    "Number of times need to be positive number.");
        }

View on GitHub (pinned to 820761864e)

Solutions

1. Validate each JSON payload against the AVRO schema before producing (avro-tools or a validator in the pipeline)
2. Fix the offending message shown in the exception text (m contains the message) to match the schema
3. Align numeric/type expectations (e.g. wrap ints as long) or regenerate the schema from the actual data shape
4. Escape/quote correctly when building JSON in shell scripts; prefer generating via jq

Example fix

// before
'{"id": "42", "amount": "12.5"}'   // id must be int, amount must be double
// after
'{"id": 42, "amount": 12.5}'

// validate payload against the AVRO schema before producing
Schema schema = new Schema.Parser().parse(schemaFile);
GenericDatumReader<Object> r = new GenericDatumReader<>(schema);
new JsonDecoder(schema, json) /* throws if JSON does not match schema */;

try {
    byte[] avro = jsonToAvro(m);
} catch (RuntimeException e) {
    if (e.getMessage() != null && e.getMessage().startsWith("Cannot convert")) {
        LOG.error("Skipping malformed message: {} — {}", m, e.getMessage());
        deadLetters.add(m); // or abort the batch
    } else { throw e; }
}

When it happens

Trigger: Producing messages whose JSON does not match the AVRO schema (missing required fields, wrong types, invalid enum values), or a malformed JSON string, when CmdProduce.generateMessageBodies invokes jsonToAvro.

Common situations: Schema file updated but sample payloads not; int vs long / string vs numeric mismatches; JSON produced by jq/shell escaping gone wrong; nullable fields sent as null when the schema marks them required.

Related errors

• Invalid schema type: ${valueSchema}
• Incompatible schema: exists schema type %s, new schema type
• Unable to create Avro schema for class <pojo.getName()>
• Cannot parse sink-config
• Cannot parse source-config

AI-assisted analysis of apache/pulsar@820761864e (2026-09-06). Data as JSON: /api/errors/8853a268bb9b784b. Report an issue: GitHub.