alibaba/canal · error · RuntimeException

Error when serializing message to byte[] by ${e.getMessage()

Error message

Error when serializing message to byte[] by ${e.getMessage()}

What it means

Thrown by CanalMessageSerializerUtil.serializer when an exception escapes the protobuf serialization of a Message to byte[]. The method computes sizes and writes a CanalPacket.Packet (MESSAGES type) via protobuf CodedOutputStream; any protobuf failure (oversized message exceeding int32/varint limits, malformed ByteString, negative size) is wrapped as a RuntimeException carrying the underlying message.

Source

Thrown at connector/core/src/main/java/com/alibaba/otter/canal/connector/core/util/CanalMessageSerializerUtil.java:82

                        for (CanalEntry.Entry entry : data.getEntries()) {
                            if (filterTransactionEntry
                                && (entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONBEGIN || entry.getEntryType() == CanalEntry.EntryType.TRANSACTIONEND)) {
                                continue;
                            }

                            messageBuilder.addMessages(entry.toByteString());
                        }

                        CanalPacket.Packet.Builder packetBuilder = CanalPacket.Packet.newBuilder();
                        packetBuilder.setType(PacketType.MESSAGES);
                        packetBuilder.setVersion(1);
                        packetBuilder.setBody(messageBuilder.build().toByteString());
                        return packetBuilder.build().toByteArray();
                    }
                }
            }
        } catch (Exception e) {
            throw new RuntimeException("Error when serializing message to byte[] by " + e.getMessage() , e);
        }
        return null;
    }

    public static Message deserializer(byte[] data) {
        return deserializer(data, false);
    }

    public static Message deserializer(byte[] data, boolean lazyParseEntry) {
        try {
            if (data == null) {
                return null;
            } else {
                CanalPacket.Packet p = CanalPacket.Packet.parseFrom(data);
                switch (p.getType()) {
                    case MESSAGES: {
                        if (!p.getCompression().equals(CanalPacket.Compression.NONE)
                            && !p.getCompression().equals(CanalPacket.Compression.COMPRESSIONCOMPATIBLEPROTO2)) {

View on GitHub (pinned to 87be50e876)

Solutions

Reduce the batch/transaction size so the serialized Message stays well under protobuf limits (tune canal.instance.memory.raw.entries / fetch sizes).
Inspect the wrapped cause (getCause()) to confirm it is a protobuf size/encoding error vs. a corrupt entry.
Validate data.isRaw()/entries are non-null and well-formed before serializing.
If persisting to MQ, confirm the broker max-message-size is not exceeded by the produced payload.

Example fix

// before: huge single batch
List<Entry> all = drainEverything();
serializer(new Message(batchId, all), false);
// after: chunk the batch
for (List<Entry> chunk : partition(all, 1000)) {
    serializer(new Message(batchId, chunk), false);
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Estimate serialized size before serializing; bail out early if oversized
long approx = data.getId() != null ? 8L : 0L;
for (Object e : data.isRaw() ? data.getRawEntries() : data.getEntries()) {
    approx += (e instanceof com.google.protobuf.ByteString)
        ? ((com.google.protobuf.ByteString) e).size() : 4096L;
}
if (approx > MAX_SAFE_MESSAGE_BYTES) {
    throw new IllegalStateException("message too large to serialize: " + approx);
}

Try / catch

try {
    byte[] out = CanalMessageSerializerUtil.serializer(data, filterTx);
} catch (RuntimeException e) {
    if (e.getMessage().startsWith("Error when serializing")) {
        Throwable c = e.getCause();
        // reduce batch size or fix corrupt entry; do not retry unchanged
    }
    throw e;
}

Prevention

Cap batch/transaction sizes fed to the serializer.
Validate entries are non-null and well-formed before serializing.
Align broker max-message-size with your batch budget.

When it happens

Trigger: A Message whose raw entries aggregate to a size exceeding protobuf's max message/field size; a corrupt ByteString in data.getRawEntries(); a too-large batch pushed to the MQ producer.

Common situations: Very large transactions producing a single Message that exceeds protobuf's ~2GB/int32 ceiling; corrupted in-memory entry from a parser bug; feeding a Message with null/invalid entries into serializer.

Related errors

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/e4f63eb213adb8c7. Report an issue: GitHub.