{"record":{"id":"8853a268bb9b784b","repo":"apache/pulsar","slug":"cannot-convert-m-to-avro-e-getmessage","errorCode":null,"errorMessage":"Cannot convert ${m} to AVRO ${e.getMessage()}","messagePattern":"Cannot convert (.+?) to AVRO (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/CmdProduce.java","lineNumber":231,"sourceCode":"        try {\n            GenericDatumReader<Object> reader = new GenericDatumReader<>(avroSchema);\n            JsonDecoder jsonDecoder = DecoderFactory.get().jsonDecoder(avroSchema, m);\n            GenericDatumWriter<Object> writer = new GenericDatumWriter<>(avroSchema);\n            ByteArrayOutputStream out = new ByteArrayOutputStream();\n            Encoder e = EncoderFactory.get().binaryEncoder(out, null);\n            Object datum = null;\n            while (true) {\n                try {\n                    datum = reader.read(datum, jsonDecoder);\n                } catch (EOFException eofException) {\n                    break;\n                }\n                writer.write(datum, e);\n                e.flush();\n            }\n            return out.toByteArray();\n        } catch (IOException e) {\n            throw new RuntimeException(\"Cannot convert \" + m + \" to AVRO \" + e.getMessage(), e);\n        }\n    }\n\n    @Spec\n    private CommandSpec commandSpec;\n\n    /**\n     * Run the producer.\n     *\n     * @return 0 for success, < 0 otherwise\n     * @throws Exception\n     */\n    @SuppressWarnings({\"rawtypes\", \"unchecked\"})\n    public int run() throws PulsarClientException {\n        if (this.numTimesProduce <= 0) {\n            throw new CommandLine.ParameterException(commandSpec.commandLine(),\n                    \"Number of times need to be positive number.\");\n        }","sourceCodeStart":213,"sourceCodeEnd":249,"githubUrl":"https://github.com/apache/pulsar/blob/820761864ed8e2a7d2e52dd9763ad2ae117c1395/pulsar-client-tools/src/main/java/org/apache/pulsar/client/cli/CmdProduce.java#L213-L249","documentation":"jsonToAvro converts each JSON message body into AVRO-encoded bytes using an Avro writer; any IOException during encoding/flushing is rethrown as a RuntimeException 'Cannot convert <message> to AVRO <reason>'. It means one of the supplied message payloads could not be encoded against the configured AVRO schema.","triggerScenarios":"Producing messages whose JSON does not match the AVRO schema (missing required fields, wrong types, invalid enum values), or a malformed JSON string, when CmdProduce.generateMessageBodies invokes jsonToAvro.","commonSituations":"Schema file updated but sample payloads not; int vs long / string vs numeric mismatches; JSON produced by jq/shell escaping gone wrong; nullable fields sent as null when the schema marks them required.","solutions":["Validate each JSON payload against the AVRO schema before producing (avro-tools or a validator in the pipeline)","Fix the offending message shown in the exception text (m contains the message) to match the schema","Align numeric/type expectations (e.g. wrap ints as long) or regenerate the schema from the actual data shape","Escape/quote correctly when building JSON in shell scripts; prefer generating via jq"],"exampleFix":"// before\n'{\"id\": \"42\", \"amount\": \"12.5\"}'   // id must be int, amount must be double\n// after\n'{\"id\": 42, \"amount\": 12.5}'","handlingStrategy":"try-catch","validationCode":"// validate payload against the AVRO schema before producing\nSchema schema = new Schema.Parser().parse(schemaFile);\nGenericDatumReader<Object> r = new GenericDatumReader<>(schema);\nnew JsonDecoder(schema, json) /* throws if JSON does not match schema */;","typeGuard":null,"tryCatchPattern":"try {\n    byte[] avro = jsonToAvro(m);\n} catch (RuntimeException e) {\n    if (e.getMessage() != null && e.getMessage().startsWith(\"Cannot convert\")) {\n        LOG.error(\"Skipping malformed message: {} — {}\", m, e.getMessage());\n        deadLetters.add(m); // or abort the batch\n    } else { throw e; }\n}","preventionTips":["Run avro-tools or a CI validator to check sample payloads against the schema before producing","Regenerate/verify the schema whenever message fields or types change","Avoid hand-built JSON in shell; use jq or a templating tool with proper escaping","Wrap jsonToAvro per-message so one bad payload doesn't abort the whole produce run"],"tags":["avro","json","serialization","cli"],"backgroundTag":"avro-schema-mismatch","analyzedSha":"820761864ed8e2a7d2e52dd9763ad2ae117c1395","analyzedAt":"2026-09-06T00:14:20.138Z","contentChangedAt":"2026-09-06T00:14:20.138Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}