What causes "fetch finished actors mismatch: expected: {:?}, actual: {:?}" in risingwavelabs/risingwave?

Compare the logged expected vs actual fetch-finished actor sets to find the offending ids and the pass that recorded them.

risingwavelabs/risingwave · error

fetch finished actors mismatch: expected: {:?}, actual: {:?}

Error message

fetch finished actors mismatch: expected: {:?}, actual: {:?}

What it means

`is_load_finished` mirrors the list-phase check for the fetch/load phase: when the number of fetch-finished actors reaches the expected count, the actual `fetch_finished_actors` set must equal `expected_fetch_actors`. If not, the recorded completion set has drifted from the plan, and the manager throws this error rather than reporting the load phase as done. It protects against completing a refresh on partially or wrongly tracked actor completion.

Source

Thrown at src/meta/src/stream/refresh_manager.rs:567

                    self.expected_list_actors,
                    self.list_finished_actors
                )))
            }
        } else {
            Ok(false)
        }
    }

    pub fn report_load_finished(&mut self, actor_ids: impl Iterator<Item = ActorId>) {
        self.fetch_finished_actors.extend(actor_ids);
    }

    pub fn is_load_finished(&self) -> MetaResult<bool> {
        if self.fetch_finished_actors.len() >= self.expected_fetch_actors.len() {
            if self.expected_fetch_actors == self.fetch_finished_actors {
                Ok(true)
            } else {
                Err(MetaError::from(anyhow!(
                    "fetch finished actors mismatch: expected: {:?}, actual: {:?}",
                    self.expected_fetch_actors,
                    self.fetch_finished_actors
                )))
            }
        } else {
            Ok(false)
        }
    }
}

View on GitHub (pinned to 6469eb736d)

Solutions

Compare the logged expected vs actual fetch-finished actor sets to find the offending ids and the pass that recorded them.
Retry the refresh so the manager is recreated with a coherent expected set.
Ensure no concurrent reschedule mutates expected_fetch_actors while fetch is in progress.
If reproducible, file a meta bug: this is an internal bookkeeping inconsistency, not operator error.

Defensive patterns

Strategy: try-catch

Try / catch

match manager.is_load_finished() {
    Ok(true) => complete_refresh(),
    Ok(false) => keep_polling(),
    Err(e) => { warn!("fetch bookkeeping drift: {e}"); rebuild_manager_and_retry(); }
}

Prevention

Serialize refresh and reschedule operations on the same job.
Discard stale fetch-finish notifications after any plan change.
Treat the expected set as immutable during a refresh round.
Add logging of expected vs actual sets on every poll to ease diagnosis.

When it happens

Trigger: Calling `is_load_finished` when `fetch_finished_actors.len() >= expected_fetch_actors.len()` but the sets differ — e.g. notifications for actors no longer in the expected plan, duplicates counted via len(), or the expected plan was updated mid-flight.

Common situations: A reschedule changed the actor layout while old fetch-finish reports were still queued; a source/backfill actor reported completion out of band; repeated refresh invocations sharing stale state.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/08c48bce906cd27c. Report an issue: GitHub.