risingwavelabs/risingwave · error

invalid backfill state: backfill_finished

Error message

invalid backfill state: backfill_finished

What it means

Same CDC backfill state schema; `restore_state` requires the `backfill_finished` column (third from the end) to be a non-NULL Bool. A NULL or non-bool value means the persisted state cannot be interpreted, so recovery bails with this error.

Source

Thrown at src/stream/src/executor/backfill/cdc/state.rs:78

        let key = Some(self.split_id.clone());
        match self
            .state_table
            .get_row(row::once(key.map(ScalarImpl::from)))
            .await?
        {
            Some(row) => {
                tracing::info!("restored cdc backfill state: {:?}", row);
                self.cached_state = row.into_inner().into_vec();
                let state = self.cached_state.as_slice();
                let state_len = state.len();
                // schema: | `split_id` | `pk...` | `backfill_finished` | `row_count` | `cdc_offset` |
                let row_count = match state[state_len - 2] {
                    Some(ScalarImpl::Int64(val)) => val,
                    _ => return Err(anyhow!("invalid backfill state: row_count").into()),
                };
                let is_finished = match state[state_len - 3] {
                    Some(ScalarImpl::Bool(val)) => val,
                    _ => return Err(anyhow!("invalid backfill state: backfill_finished").into()),
                };
                let cdc_offset = match state[state_len - 1] {
                    Some(ScalarImpl::Jsonb(ref jsonb)) => {
                        serde_json::from_value(jsonb.clone().take()).unwrap()
                    }
                    None if is_finished => None,
                    None => {
                        return Err(anyhow!(
                            "invalid backfill state: unfinished row has null cdc_offset"
                        )
                        .into());
                    }
                    _ => return Err(anyhow!("invalid backfill state: cdc_offset").into()),
                };

                let current_pk_pos = state[1..state_len - 3].to_vec();
                Ok(CdcStateRecord {
                    current_pk_pos: Some(OwnedRow::new(current_pk_pos)),

View on GitHub (pinned to 6469eb736d)

Solutions

1. Re-create the CDC table / backfilled object so its state table is rebuilt from the beginning.
2. Check version compatibility between the build that wrote the state and the current one; rebuild after upgrades.
3. Dump the state row and verify the column layout matches `split_id | pk | backfill_finished | row_count | cdc_offset`.
4. Restore from backup of the state store if the data is valuable and corruption is confirmed.

When it happens

Trigger: Recovery of a CDC backfill executor when the state row's `backfill_finished` slot is NULL or holds a non-boolean scalar — incomplete write before crash, schema drift, or corruption.

Common situations: Crash between partial state writes during CDC backfill, upgrades with changed state layout, or corrupted Hummock state rows.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• invalid backfill state: row_count
• invalid backfill state: unfinished row has null cdc_offset
• ParallelizedCdcBackfillExecutor expects either Mutation::Add
• legacy no-shuffle backfill recovered unfinished progress; ca
• implement MySQL CDC parallelized backfill

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/347d4084ca005fbf. Report an issue: GitHub.