progress table column not matched with upstream table schema

Error message

progress table column not matched with upstream table schema: progress table: {:?}, pk: {:?}

What it means

The progress table's column types do not match, position by position, the expected sequence of `EXTRA_COLUMN_TYPES` followed by the upstream table's PK column types. Same schema-contract as the length check, but at the type level.

Source

Thrown at src/stream/src/executor/backfill/snapshot_backfill/state.rs:72

        upstream_pk_column_types: &[DataType],
    ) -> StreamExecutorResult<()> {
        if progress_table_column_types.len()
            != EXTRA_COLUMN_TYPES.len() + upstream_pk_column_types.len()
        {
            return Err(anyhow!(
                "progress table columns len not matched with the len derived from upstream table pk. progress table: {:?}, pk: {:?}",
                progress_table_column_types,
                upstream_pk_column_types)
                .into()
            );
        }
        for (expected_type, progress_table_type) in EXTRA_COLUMN_TYPES
            .iter()
            .chain(upstream_pk_column_types.iter())
            .zip_eq_debug(progress_table_column_types.iter())
        {
            if expected_type != progress_table_type {
                return Err(anyhow!(
                    "progress table column not matched with upstream table schema: progress table: {:?}, pk: {:?}",
                    progress_table_column_types,
                    upstream_pk_column_types)
                    .into()
                );
            }
        }
        Ok(())
    }

    pub(super) fn from_row(row: &OwnedRow, pk_serde: &OrderedRowSerde) -> Self {
        assert_eq!(
            row.len(),
            pk_serde.get_data_types().len() + EXTRA_COLUMN_TYPES.len() - 1, /* Pk of the progress state table (i.e. vnode column) not included */
        );
        let epoch = must_match!(&row[0], Some(ScalarImpl::Int64(epoch)) => {
           *epoch as u64
        });

View on GitHub (pinned to 6469eb736d)

Solutions

Recreate the target MV/table so a fresh progress table is built from the current upstream schema.
Diff the actual progress table column types against `EXTRA_COLUMN_TYPES` + upstream PK types and align the source of drift.
Verify no manual schema alteration was applied to the progress or source table.

Defensive patterns

Strategy: validation

Validate before calling

for (expected, actual) in EXTRA_COLUMN_TYPES.iter().chain(pk_types.iter()).zip(pt_types.iter()) { if expected != actual { return Err(anyhow!("type mismatch: {expected:?} vs {actual:?}")); } }

Try / catch

if let Err(e) = BackfillState::validate_progress_table_schema(&pt, &pk) { drop_and_recreate_state(); return Err(e); }

Prevention

Diff progress table types against the upstream PK after any source schema change
Avoid restoring progress tables from mismatched backups
Run schema validation as a preflight during backfill state recovery

When it happens

Trigger: A progress table whose columns have different `DataType`s than expected — e.g. PK column type changed upstream (INT → BIGINT), or the progress table was created with legacy/different types.

Common situations: Schema drift between the source table and the persisted progress table; restoring state from backup of a different table definition; version upgrades that changed type mapping.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/a701822ea3c7eca9. Report an issue: GitHub.