risingwavelabs/risingwave · error

list finished actors mismatch: expected: {:?}, actual: {:?}

Error message

list finished actors mismatch: expected: {:?}, actual: {:?}

What it means

In the streaming job refresh manager, `is_list_finished` validates that the set of actors whose list phase finished (`list_finished_actors`) exactly equals the expected set (`expected_list_actors`). Once the reported count reaches the expected count, the two sets are compared; any divergence (extra, missing, or wrong actor ids) means the refresh bookkeeping is inconsistent, so the meta service raises this error instead of declaring the phase complete. It is an internal consistency check on reschedule/refresh progress.

Source

Thrown at src/meta/src/stream/refresh_manager.rs:547

        Self {
            expected_list_actors: HashSet::new(),
            expected_fetch_actors: HashSet::new(),
            list_finished_actors: HashSet::new(),
            fetch_finished_actors: HashSet::new(),
            start_time: Instant::now(),
        }
    }

    pub fn report_list_finished(&mut self, actor_ids: impl Iterator<Item = ActorId>) {
        self.list_finished_actors.extend(actor_ids);
    }

    pub fn is_list_finished(&self) -> MetaResult<bool> {
        if self.list_finished_actors.len() >= self.expected_list_actors.len() {
            if self.expected_list_actors == self.list_finished_actors {
                Ok(true)
            } else {
                Err(MetaError::from(anyhow!(
                    "list finished actors mismatch: expected: {:?}, actual: {:?}",
                    self.expected_list_actors,
                    self.list_finished_actors
                )))
            }
        } else {
            Ok(false)
        }
    }

    pub fn report_load_finished(&mut self, actor_ids: impl Iterator<Item = ActorId>) {
        self.fetch_finished_actors.extend(actor_ids);
    }

    pub fn is_load_finished(&self) -> MetaResult<bool> {
        if self.fetch_finished_actors.len() >= self.expected_fetch_actors.len() {
            if self.expected_fetch_actors == self.fetch_finished_actors {
                Ok(true)

View on GitHub (pinned to 6469eb736d)

Solutions

1. Inspect the logged expected vs actual actor id sets to identify the stray or missing actor and trace which reschedule pass recorded it.
2. Retry the refresh/reschedule operation so the manager is rebuilt from a consistent expected-actor snapshot.
3. Check for concurrent reschedules/refreshes on the same table/job and serialize them.
4. If reproducible, report as a meta bug — the mismatch implies broken bookkeeping in the refresh manager rather than user error.

Example fix

// before: recording finish notifications without matching against the current expected set
manager.record_list_finished(actor_id);
// after: only record ids that belong to the current expected plan
if manager.expected_list_actors().contains(&actor_id) {
    manager.record_list_finished(actor_id);
}

// before calling, ensure no reschedule is in flight and the manager was freshly built
assert!(expected.iter().all(|a| !reported.contains(a)) || expected == reported);

match manager.is_list_finished() {
    Ok(true) => proceed_to_next_phase(),
    Ok(false) => keep_polling(),
    Err(e) => log_error_and_retry_refresh(e), // rebuild the refresh manager and retry
}

When it happens

Trigger: Calling `is_list_finished` on a `RefreshManager` when `list_finished_actors.len() >= expected_list_actors.len()` but the two sets differ — e.g. duplicate or stale actor ids were recorded as list-finished, or the expected set changed (new reschedule) while in-flight notifications from the old plan kept arriving.

Common situations: Race during a reschedule where the expected actor list is regenerated while finish notifications for old actors are still being recorded; a bug in the notification path reporting the wrong actor ids; concurrent refresh attempts sharing the same manager state.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• fetch finished actors mismatch: expected: {:?}, actual: {:?}
• schema_id and partition_spec_id should be the same in all wr
• reschedule failed
• duplicate worker id {worker_id} in plan, prev {worker_id} ->
• inconsistent hummock version: expected {}, actual {}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/e9aef98ce9cef713. Report an issue: GitHub.