risingwavelabs/risingwave · critical

worker_id {} for actor {} does not exist

Error message

worker_id {} for actor {} does not exist

What it means

validate_database_info (recovery validation) checks that every streaming actor's assigned worker_id exists among the active streaming compute nodes. If an actor references a worker not in active_streaming_nodes, recovery aborts with this error, indicating inconsistent cluster metadata between the job's fragment graph and the live worker set.

Source

Thrown at src/meta/src/barrier/mod.rs:169

    cdc_table_snapshot_splits: HashMap<JobId, CdcTableSnapshotSplits>,
}

impl BarrierWorkerRuntimeInfoSnapshot {
    fn validate_database_info(
        database_id: DatabaseId,
        database_jobs: &HashMap<JobId, HashMap<FragmentId, InflightFragmentInfo>>,
        active_streaming_nodes: &ActiveStreamingWorkerNodes,
        stream_actors: &HashMap<ActorId, StreamActor>,
        state_table_committed_epochs: &HashMap<TableId, u64>,
    ) -> MetaResult<()> {
        {
            for fragment in database_jobs.values().flat_map(|job| job.values()) {
                for (actor_id, actor) in &fragment.actors {
                    if !active_streaming_nodes
                        .current()
                        .contains_key(&actor.worker_id)
                    {
                        return Err(anyhow!(
                            "worker_id {} for actor {} does not exist",
                            actor.worker_id,
                            actor_id
                        )
                        .into());
                    }
                    if !stream_actors.contains_key(actor_id) {
                        return Err(anyhow!("cannot find StreamActor of actor {}", actor_id).into());
                    }
                }
                for state_table_id in &fragment.state_table_ids {
                    if !state_table_committed_epochs.contains_key(state_table_id) {
                        return Err(anyhow!(
                            "state table {} is not registered to hummock",
                            state_table_id
                        )
                        .into());
                    }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure the worker is running and registered so active_streaming_nodes includes it, then retry recovery
2. If the worker is permanently gone, reschedule/rebuild affected jobs so actors map to live workers
3. Check metadata consistency (actor->worker mapping) and repair stale fragment metadata
4. Report as a bug if actor placement produced a worker_id that never existed

// before recovery-sensitive operations, confirm all actor worker_ids are live
for actor in all_actors() {
    if !active_nodes.contains_key(&actor.worker_id) {
        return Err(format!("worker {} for actor {} missing", actor.worker_id, actor.actor_id));
    }
}

fn is_worker_active(worker_id: u32, nodes: &HashMap<u32, WorkerNode>) -> bool { nodes.contains_key(&worker_id) }

match validate_database_info(...) {
    Err(e) if e.to_string().contains("does not exist") => {
        // reschedule actors of missing workers, then retry recovery
        reschedule_actors_of_missing_workers();
        retry_recovery().await;
    }
    Err(e) => return Err(e),
    Ok(v) => v,
}

When it happens

Trigger: Recovery where an actor's worker_id is absent from active streaming nodes — e.g. a compute node was removed and its actors were not reassigned, or stale worker mapping in the fragment metadata.

Common situations: Compute node scale-in without proper actor rescheduling; worker crash between metadata snapshots; recovery from an inconsistent metadata checkpoint.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

• job {} in database {} has no state table after recovery
• job {} in database {} has tables with different table ids. {
• inconsistent hummock version: expected {}, actual {}
• since_timestamp requires at least one upstream table
• The cluster is recovering

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/eb4be97536ee396a. Report an issue: GitHub.