risingwavelabs/risingwave · error

invalid backfill state: cdc_offset

Error message

invalid backfill state: cdc_offset

What it means

Thrown by CdcBackfillState::restore_state when the persisted state row for a CDC backfill has an unexpected shape: the cdc_offset element in the state array is neither a valid value nor the expected null. The state table row layout (finished flag, offsets, PK position) does not match what the executor version expects, so recovery is refused rather than silently resuming with corrupt progress.

Source

Thrown at src/stream/src/executor/backfill/cdc/state.rs:91

                    Some(ScalarImpl::Int64(val)) => val,
                    _ => return Err(anyhow!("invalid backfill state: row_count").into()),
                };
                let is_finished = match state[state_len - 3] {
                    Some(ScalarImpl::Bool(val)) => val,
                    _ => return Err(anyhow!("invalid backfill state: backfill_finished").into()),
                };
                let cdc_offset = match state[state_len - 1] {
                    Some(ScalarImpl::Jsonb(ref jsonb)) => {
                        serde_json::from_value(jsonb.clone().take()).unwrap()
                    }
                    None if is_finished => None,
                    None => {
                        return Err(anyhow!(
                            "invalid backfill state: unfinished row has null cdc_offset"
                        )
                        .into());
                    }
                    _ => return Err(anyhow!("invalid backfill state: cdc_offset").into()),
                };

                let current_pk_pos = state[1..state_len - 3].to_vec();
                Ok(CdcStateRecord {
                    current_pk_pos: Some(OwnedRow::new(current_pk_pos)),
                    is_finished,
                    last_cdc_offset: cdc_offset,
                    row_count,
                })
            }
            None => Ok(CdcStateRecord::default()),
        }
    }

    /// Modify the state of the corresponding split (currently only supports single split)
    pub async fn mutate_state(
        &mut self,
        current_pk_pos: Option<OwnedRow>,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Cancel and recreate the streaming job (MV/materialized view with CDC source) so backfill restarts from a clean state
2. Check the RisingWave version the state was written with; roll back to it, let the backfill finish, then upgrade
3. Inspect the state table row (SELECT the internal state table) to confirm the row layout mismatch
4. If corruption is isolated, drop and rebuild the affected table/backfill from the upstream CDC source

// Before relying on recovery, inspect the persisted state row types
let state = state_store.get_row(...)?.into_inner().into_vec();
let cdc_offset_ok = matches!(state.get(STATE_CDC_OFFSET_IDX), None | Some(ScalarImpl::Bytea(_)));
if !cdc_offset_ok { /* recreate the job instead of resuming */ }

When it happens

Trigger: Restoring a CDC backfill actor whose state table row contains a scalar at the cdc_offset position with an unexpected type (neither a valid offset nor None), e.g. after a partial write, a state-schema change, or reading a row written by an incompatible version.

Common situations: Upgrading RisingWave across versions that changed the CDC backfill state row layout; a state table row written by an old migration that did not include the cdc_offset field; manual state-table edits or corrupted persisted state.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• invalid backfill state: backfill_finished
• invalid backfill state: row_count
• invalid backfill state: cdc_offset_low
• invalid backfill state: cdc_offset_high
• next offset {:?} should be later than current offset {:?}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/6c0ce968ac73bfd6. Report an issue: GitHub.