What causes "failed to parse {} row(s) in a single canal json message: {}" in risingwavelabs/risingwave?

Read the joined inner error messages to identify the offending rows/columns

failed to parse {} row(s) in a single canal json message: {}

Error message

failed to parse {} row(s) in a single canal json message: {}

What it means

A single Canal JSON message may carry multiple rows in its `data` array. Rows are parsed individually and errors collected; if any row fails, the whole message fails with this aggregated error listing the count and all underlying row errors.

Source

Thrown at src/connector/src/parser/canal/simd_json_parser.rs:101

                BorrowedValue::Array(array) => Some(array),
                _ => None,
            })
            .context("field `data` is missing for creating event")?;

        let mut errors = Vec::new();
        for event in events.drain(..) {
            let accessor = JsonAccess::new_with_options(event, &JsonParseOptions::CANAL);
            match apply_row_operation_on_stream_chunk_writer((op, accessor), &mut writer) {
                Ok(_) => {}
                Err(err) => errors.push(err),
            }
        }

        if errors.is_empty() {
            Ok(())
        } else {
            // TODO(error-handling): multiple errors
            bail!(
                "failed to parse {} row(s) in a single canal json message: {}",
                errors.len(),
                errors.iter().format(", ")
            );
        }
    }
}

impl ByteStreamSourceParser for CanalJsonParser {
    fn columns(&self) -> &[SourceColumnDesc] {
        &self.rw_columns
    }

    fn source_ctx(&self) -> &SourceContext {
        &self.source_ctx
    }

    fn parser_format(&self) -> ParserFormat {

View on GitHub (pinned to 6469eb736d)

Solutions

Read the joined inner error messages to identify the offending rows/columns
Align the source schema with the MySQL table (recreate source or alter columns) after schema changes upstream
Fix or sanitize the upstream data causing the per-row failures
If partial ingestion is acceptable, split upstream messages to single-row events so only the bad row fails

Defensive patterns

Strategy: try-catch

Validate before calling

// pre-validate rows against expected columns
for (const row of msg.data) { for (const col of requiredColumns) if (!(col in row)) throw new Error(`row missing column ${col}`); }

Try / catch

match res { Err(e) if e.to_string().starts_with("failed to parse") => { log_full(e); route_to_dlq(msg); } , Err(e) => propagate(e), Ok(v) => v }

Prevention

Keep MySQL table schema and source schema in sync; run schema-drift checks
Split large multi-row binlog events upstream to isolate bad rows
Use a DLQ for unparseable messages instead of stalling the source

When it happens

Trigger: Any of the row-level parse failures (column mismatch, JSON value type not convertible to the target column type, etc.) in a multi-row `data` array.

Common situations: Schema drift between MySQL table and the RW source columns (added/renamed/retyped columns); bad data types inserted upstream (e.g. oversized strings into SMALLINT); messages produced with multiple rows per binlog event.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/20880d44f6f18c9e. Report an issue: GitHub.