{"record":{"id":"e4f63eb213adb8c7","repo":"alibaba/canal","slug":"error-when-serializing-message-to-byte-by-e-ge","errorCode":null,"errorMessage":"Error when serializing message to byte[] by ${e.getMessage()}","messagePattern":"Error when serializing message to byte\\[\\] by (.+?)","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"connector/core/src/main/java/com/alibaba/otter/canal/connector/core/util/CanalMessageSerializerUtil.java","lineNumber":82,"sourceCode":"                        for (CanalEntry.Entry entry : data.getEntries()) {\n                            if (filterTransactionEntry\n                                && (entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONBEGIN || entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONEND)) {\n                                continue;\n                            }\n\n                            messageBuilder.addMessages(entry.toByteString());\n                        }\n\n                        CanalPacket.Packet.Builder packetBuilder = CanalPacket.Packet.newBuilder();\n                        packetBuilder.setType(PacketType.MESSAGES);\n                        packetBuilder.setVersion(1);\n                        packetBuilder.setBody(messageBuilder.build().toByteString());\n                        return packetBuilder.build().toByteArray();\n                    }\n                }\n            }\n        } catch (Exception e) {\n            throw new RuntimeException(\"Error when serializing message to byte[] by \" + e.getMessage() , e);\n        }\n        return null;\n    }\n\n    public static Message deserializer(byte[] data) {\n        return deserializer(data, false);\n    }\n\n    public static Message deserializer(byte[] data, boolean lazyParseEntry) {\n        try {\n            if (data == null) {\n                return null;\n            } else {\n                CanalPacket.Packet p = CanalPacket.Packet.parseFrom(data);\n                switch (p.getType()) {\n                    case MESSAGES: {\n                        if (!p.getCompression().equals(CanalPacket.Compression.NONE)\n                            && !p.getCompression().equals(CanalPacket.Compression.COMPRESSIONCOMPATIBLEPROTO2)) {","sourceCodeStart":64,"sourceCodeEnd":100,"githubUrl":"https://github.com/alibaba/canal/blob/87be50e87686a3e8af08c368d0e1ffd1f59eb04a/connector/core/src/main/java/com/alibaba/otter/canal/connector/core/util/CanalMessageSerializerUtil.java#L64-L100","documentation":"Thrown by CanalMessageSerializerUtil.serializer when an exception escapes the protobuf serialization of a Message to byte[]. The method computes sizes and writes a CanalPacket.Packet (MESSAGES type) via protobuf CodedOutputStream; any protobuf failure (oversized message exceeding int32/varint limits, malformed ByteString, negative size) is wrapped as a RuntimeException carrying the underlying message.","triggerScenarios":"A Message whose raw entries aggregate to a size exceeding protobuf's max message/field size; a corrupt ByteString in data.getRawEntries(); a too-large batch pushed to the MQ producer.","commonSituations":"Very large transactions producing a single Message that exceeds protobuf's ~2GB/int32 ceiling; corrupted in-memory entry from a parser bug; feeding a Message with null/invalid entries into serializer.","solutions":["Reduce the batch/transaction size so the serialized Message stays well under protobuf limits (tune canal.instance.memory.raw.entries / fetch sizes).","Inspect the wrapped cause (getCause()) to confirm it is a protobuf size/encoding error vs. a corrupt entry.","Validate data.isRaw()/entries are non-null and well-formed before serializing.","If persisting to MQ, confirm the broker max-message-size is not exceeded by the produced payload."],"exampleFix":"// before: huge single batch\nList<Entry> all = drainEverything();\nserializer(new Message(batchId, all), false);\n// after: chunk the batch\nfor (List<Entry> chunk : partition(all, 1000)) {\n    serializer(new Message(batchId, chunk), false);\n}","handlingStrategy":"try-catch","validationCode":"// Estimate serialized size before serializing; bail out early if oversized\nlong approx = data.getId() != null ? 8L : 0L;\nfor (Object e : data.isRaw() ? data.getRawEntries() : data.getEntries()) {\n    approx += (e instanceof com.google.protobuf.ByteString)\n        ? ((com.google.protobuf.ByteString) e).size() : 4096L;\n}\nif (approx > MAX_SAFE_MESSAGE_BYTES) {\n    throw new IllegalStateException(\"message too large to serialize: \" + approx);\n}","typeGuard":null,"tryCatchPattern":"try {\n    byte[] out = CanalMessageSerializerUtil.serializer(data, filterTx);\n} catch (RuntimeException e) {\n    if (e.getMessage().startsWith(\"Error when serializing\")) {\n        Throwable c = e.getCause();\n        // reduce batch size or fix corrupt entry; do not retry unchanged\n    }\n    throw e;\n}","preventionTips":["Cap batch/transaction sizes fed to the serializer.","Validate entries are non-null and well-formed before serializing.","Align broker max-message-size with your batch budget."],"tags":["serialization","protobuf","mq","message-size"],"backgroundTag":null,"analyzedSha":"87be50e87686a3e8af08c368d0e1ffd1f59eb04a","analyzedAt":"2026-08-14T04:30:11.918Z","schemaVersion":2},"datasetVersion":"2026-08-14T05:17:29.042Z"}