risingwavelabs/risingwave · error

Debezium Mongo needs a `_id` column in table

Error message

Debezium Mongo needs a `_id` column in table

What it means

`DebeziumMongoJsonParser::new` validates that the target table schema is compatible with Debezium MongoDB CDC. Debezium emits MongoDB documents whose `_id` field is the key, so RisingWave requires exactly the `_id` column plus the two auto-generated offset/file columns (`_rw_{connector}_file` / `_rw_{connector}_offset`), i.e. exactly 2 visible non-additional columns including `_id`. Anything else is rejected at parser construction.

Source

Thrown at src/connector/src/parser/debezium/mongo_json_parser.rs:84

            .context("Debezium Mongo needs a `_id` column with supported types (Varchar Jsonb int32 int64) in table")?.clone();

        if !props.strong_schema {
            let _payload_column = rw_columns
                .iter()
                .find(|desc| desc.name == "payload" && matches!(desc.data_type, DataType::Jsonb))
                .context(
                    "Debezium Mongo needs a `payload` column with supported types Jsonb in table",
                )?
                .clone();

            let columns = rw_columns
                .iter()
                .filter(|desc| desc.is_visible() && desc.additional_column.column_type.is_none())
                .count();

            // _rw_{connector}_file/partition & _rw_{connector}_offset are created automatically.
            if columns != 2 || !rw_columns.iter().any(|desc| desc.name == "_id") {
                bail!("Debezium Mongo needs a `_id` column in table");
            }
        }

        // encodings are fixed to MongoJson
        let encoding = EncodingProperties::MongoJson(props);
        // for key, it doesn't matter if strong schema is enabled or not
        let key_builder = build_accessor_builder(encoding.clone())?;

        let payload_builder = build_accessor_builder(encoding)?;

        Ok(Self {
            rw_columns,
            source_ctx,
            key_builder,
            payload_builder,
        })
    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Add an `_id` column to the source table definition (e.g. `_id JSONB PRIMARY KEY`).
2. Remove extra visible columns so only `_id` and the auto offset column remain; project extra fields downstream instead.
3. Confirm no additional_column types are attached that change the visible column count, and re-check `is_visible()` on each column desc.

Example fix

-- before
CREATE TABLE t (user_id JSONB, payload JSONB) WITH (
  connector='mongodb-cdc', ...
);
-- after
CREATE TABLE t (_id JSONB PRIMARY KEY, payload JSONB) WITH (
  connector='mongodb-cdc', ...
);

let visible: Vec<_> = rw_columns.iter().filter(|d| d.is_visible() && d.additional_column.column_type.is_none()).collect();
assert!(visible.len() == 2 && visible.iter().any(|d| d.name == "_id"), "table must define _id plus one offset column");

match res {
    Err(e) if e.to_string().contains("needs a `_id` column") => fix_table_schema_and_retry(),
    Err(e) => return Err(e),
    Ok(v) => v,
}

When it happens

Trigger: Creating a Mongo CDC source/table whose visible column list does not contain a column named `_id`, or whose count of visible columns without additional-column types is not exactly 2 (the `_id` plus one offset column).

Common situations: Declaring a Mongo CDC table with renamed or extra business columns; forgetting the `_id` column in the CREATE TABLE; migrating a schema from a non-Mongo CDC connector where the `_id` requirement doesn't exist; auto schema derivation failing to include `_id`.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Postgres table should define the primary key for non-append-
• unsupported encoding for DEBEZIUM_MONGO format
• {connector_name} snapshot primary-key index {index} is out o
• {connector_name} snapshot primary key `{}` cannot be NULL
• MySQL table doesn't define the primary key

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/577271a9d489cc4b. Report an issue: GitHub.