alibaba/canal · error · IllegalArgumentException

reading json diff

Error message

reading json diff

What it means

Thrown at the end of the first pass (skip/scan pass) of JSON diff parsing when the total bytes consumed (buffer.position() - position) does not equal the expected diff length (len). This means the operations consumed more or fewer bytes than declared, indicating the diff payload is internally inconsistent.

Source

Thrown at dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/JsonDiffConversion.java:83

            buffer.forward((int) path_length);
            // skip value
            if (operation_int != DIFF_OPERATION_REMOVE) {
                long value_length = buffer.getPackedLong();
                if (value_length > len) {
                    throw new IllegalArgumentException("skipping path");
                }

                buffer.forward((int) value_length);
            }

            // see https://github.com/alibaba/canal/pull/5018
            if (buffer.position() - position >= len) {
                break;
            }
        }

        if (buffer.position() - position != len) {
            throw new IllegalArgumentException("reading json diff");
        }

        // Print function names in reverse order.
        StringBuilder builder = new StringBuilder();
        for (int i = operation_names.size() - 1; i >= 0; i--) {
            if (i == 0 || !Objects.equals(operation_names.get(i - 1), operation_names.get(i))) {
                builder.append(operation_names.get(i)).append("(");
            }
        }

        // Print column id
        if (columnName != null) {
            builder.append(columnName);
        } else {
            builder.append("@").append(columnIndex);
        }

        // In case this vector is empty (a no-op), make an early return

View on GitHub (pinned to 87be50e876)

Solutions

1. Compare the actual consumed bytes (buffer.position() - position) with len to understand the magnitude of the discrepancy.
2. Hex-dump the full diff payload and manually verify each operation's path_length and value_length fields.
3. Verify the canal/dbsync version supports the specific MySQL 8.0 minor version generating these partial updates.
4. Re-synchronize replication from a known-good position if the binlog is genuinely corrupt.

try {
    JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charset);
} catch (IllegalArgumentException e) {
    if (e.getMessage().equals("reading json diff")) {
        long consumed = buffer.position() - startPos;
        logger.warn("JSON diff consumed {} bytes but expected {}, column={}", consumed, len, columnName);
    }
    throw e;
}

When it happens

Trigger: After the while loop in print_json_diff's first pass processes all operations (the loop breaks when position - position >= len), a final check compares position - position against len. If they differ, the diff structure is corrupt: the sum of all operation/path/value sizes does not match the declared total.

Common situations: A partial-update event where one operation's path or value length was off-by-one, binlog corruption altering a length byte, buffer misalignment, or a MySQL 8.0 minor version change in the diff encoding format.

Related errors

• reading operation type (invalid operation code)
• skipping path
• parsing json value
• No such method: '{}' @ {}
• No such field: '{}' @ {}

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/80d87cf7c4ea19eb. Report an issue: GitHub.