risingwavelabs/risingwave · error
invalid backfill state: row_count
Error message
invalid backfill state: row_count
What it means
Thrown by CdcBackfillStateV2::restore_state when element [2] of the persisted state row is not an Int64, so the row_count (backfill progress counter) cannot be decoded. It prevents resuming a backfill with an unreadable progress value.
Source
Thrown at src/stream/src/executor/backfill/cdc/state_v2.rs:74
/// Restore the backfill state from storage
pub async fn restore_state(&mut self, split_id: i64) -> StreamExecutorResult<CdcStateRecord> {
let key = Some(split_id);
match self
.state_table
.get_row(row::once(key.map(ScalarImpl::from)))
.await?
{
Some(row) => {
tracing::info!("restored cdc backfill state: {:?}", row);
let state = row.into_inner().into_vec();
let is_finished = match state[1] {
Some(ScalarImpl::Bool(val)) => val,
_ => return Err(anyhow!("invalid backfill state: backfill_finished").into()),
};
let row_count = match state[2] {
Some(ScalarImpl::Int64(val)) => val,
_ => return Err(anyhow!("invalid backfill state: row_count").into()),
};
let (cdc_offset_low, cdc_offset_high) = if !self.is_legacy_state {
let cdc_offset_low = match state[3] {
Some(ScalarImpl::Jsonb(ref jsonb)) => {
serde_json::from_value(jsonb.clone().take()).unwrap()
}
None => None,
_ => return Err(anyhow!("invalid backfill state: cdc_offset_low").into()),
};
let cdc_offset_high = match state[4] {
Some(ScalarImpl::Jsonb(ref jsonb)) => {
serde_json::from_value(jsonb.clone().take()).unwrap()
}
None => None,
_ => return Err(anyhow!("invalid backfill state: cdc_offset_high").into()),
};
(cdc_offset_low, cdc_offset_high)
} else {View on GitHub (pinned to 6469eb736d)
Solutions
- Cancel and recreate the streaming job to rebuild backfill state from scratch
- Match the executor version to the version that wrote the state row
- Inspect the internal state table row to verify the row_count field type
- Rebuild the MV/table from the CDC upstream if the state is unrecoverable
Defensive patterns
Strategy: validation
Validate before calling
let row_count_ok = matches!(state[STATE_ROW_COUNT_IDX], Some(ScalarImpl::Int64(_)));
if !row_count_ok { /* recreate the backfill job */ } Prevention
- Avoid mixing executor versions in one cluster (rolling upgrade with state compatibility checks)
- Let backfills finish before shutting down nodes abruptly
When it happens
Trigger: Restoring a v2 CDC backfill state where state[2] is None or a non-Int64 ScalarImpl, typically because the row was written with an older/different state layout.
Common situations: Version upgrades changing the state row schema; legacy state rows read by the v2 executor; state corruption from an interrupted write.
Understand the failure class
Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.
Related errors
- invalid backfill state: cdc_offset
- invalid backfill state: backfill_finished
- invalid backfill state: cdc_offset_low
- invalid backfill state: cdc_offset_high
- next offset {:?} should be later than current offset {:?}
AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11).
Data as JSON: /api/errors/0f556886babb48e1.
Report an issue: GitHub.