alibaba/canal · error · IllegalArgumentException

reading json diff

Error message

reading json diff

What it means

Thrown at the end of the first pass (skip/scan pass) of JSON diff parsing when the total bytes consumed (buffer.position() - position) does not equal the expected diff length (len). This means the operations consumed more or fewer bytes than declared, indicating the diff payload is internally inconsistent.

Solutions

1. Compare the actual consumed bytes (buffer.position() - position) with len to understand the magnitude of the discrepancy.
2. Hex-dump the full diff payload and manually verify each operation's path_length and value_length fields.
3. Verify the canal/dbsync version supports the specific MySQL 8.0 minor version generating these partial updates.
4. Re-synchronize replication from a known-good position if the binlog is genuinely corrupt.

try {
    JsonDiffConversion.print_json_diff(buffer, len, columnName, columnIndex, charset);
} catch (IllegalArgumentException e) {
    if (e.getMessage().equals("reading json diff")) {
        long consumed = buffer.position() - startPos;
        logger.warn("JSON diff consumed {} bytes but expected {}, column={}", consumed, len, columnName);
    }
    throw e;
}

When it happens

Trigger: After the while loop in print_json_diff's first pass processes all operations (the loop breaks when position - position >= len), a final check compares position - position against len. If they differ, the diff structure is corrupt: the sum of all operation/path/value sizes does not match the declared total.

Common situations: A partial-update event where one operation's path or value length was off-by-one, binlog corruption altering a length byte, buffer misalignment, or a MySQL 8.0 minor version change in the diff encoding format.

Related errors

• parsing json value
• reading operation type (invalid operation code)
• skipping path
• command : 'show binlog events limit 1' has an error! pls…
• command : 'show binlog events limit 1' has an error!

AI-assisted analysis of alibaba/canal@87be50e876 (2026-08-14). Data as JSON: /api/errors/80d87cf7c4ea19eb. Report an issue: GitHub.

Appendix: source

Thrown at dbsync/src/main/java/com/taobao/tddl/dbsync/binlog/JsonDiffConversion.java:83

            buffer.forward((int) path_length);
            // skip value
            if (operation_int != DIFF_OPERATION_REMOVE) {
                long value_length = buffer.getPackedLong();
                if (value_length > len) {
                    throw new IllegalArgumentException("skipping path");
                }

                buffer.forward((int) value_length);
            }

            // see https://github.com/alibaba/canal/pull/5018
            if (buffer.position() - position >= len) {
                break;
            }
        }

        if (buffer.position() - position != len) {
            throw new IllegalArgumentException("reading json diff");
        }

        // Print function names in reverse order.
        StringBuilder builder = new StringBuilder();
        for (int i = operation_names.size() - 1; i >= 0; i--) {
            if (i == 0 || !Objects.equals(operation_names.get(i - 1), operation_names.get(i))) {
                builder.append(operation_names.get(i)).append("(");
            }
        }

        // Print column id
        if (columnName != null) {
            builder.append(columnName);
        } else {
            builder.append("@").append(columnIndex);
        }

        // In case this vector is empty (a no-op), make an early return

View on GitHub (pinned to 87be50e876)