risingwavelabs/risingwave · critical

should exist

Error message

should exist

What it means

This is a panic from `.expect("should exist")` in `stop_handle`, thrown when the writer handle being stopped is no longer present in the `writer_handles` map. The coordinator assumes a handle id referenced during the alter-parallelism protocol is always registered, so a missing entry is an internal invariant violation rather than a recoverable error. The process/thread panics instead of returning an error.

Source

Thrown at src/meta/src/manager/sink_coordination/coordinator_worker.rs:431

                        CoordinationHandleManagerEvent::Stop
                    }
                    coordinate_request::Msg::StartRequest(_) => {
                        unreachable!("should have been handled");
                    }
                };
                Ok((handle_id, event))
            }
        }
    }

    fn vnode_bitmap(&self, handle_id: HandleId) -> &Bitmap {
        self.writer_handles[&handle_id].vnode_bitmap()
    }

    fn stop_handle(&mut self, handle_id: HandleId) -> anyhow::Result<()> {
        self.writer_handles
            .remove(&handle_id)
            .expect("should exist")
            .stop()
    }

    async fn wait_init_handles(&mut self) -> anyhow::Result<HashSet<HandleId>> {
        assert!(self.writer_handles.is_empty());
        let mut init_requests = AligningRequests::default();
        while !init_requests.aligned() {
            let (handle_id, event) = self.next_event().await?;
            let unexpected_event = match event {
                CoordinationHandleManagerEvent::NewHandle => {
                    init_requests.add_new_request(handle_id, (), self.vnode_bitmap(handle_id))?;
                    continue;
                }
                event => event.name(),
            };
            return Err(anyhow!(
                "expect new handle during init, but got {}",
                unexpected_event

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure each HandleId is only stopped once: remove it from `remaining_handles` (already asserted) and never call `alter_parallelisms` for an id that already went through Stop.
2. Verify the handle was added via `CoordinationHandleManagerEvent::NewHandle` before calling `alter_parallelisms`; check upstream registration logic.
3. If a benign double-stop is possible, replace `.expect` with `ok_or_else(|| anyhow!(...))` and return an error so the failure is diagnosable instead of a panic.
4. Capture logs of the handle lifecycle (NewHandle/Stop events) to find which handle_id double-fires.

Example fix

// before
self.writer_handles
    .remove(&handle_id)
    .expect("should exist")
    .stop()
// after
self.writer_handles
    .remove(&handle_id)
    .ok_or_else(|| anyhow!("handle {:?} does not exist when stopping", handle_id))?
    .stop()

// coordinator-side precheck
if !self.writer_handles.contains_key(&handle_id) {
    return Err(anyhow!("cannot stop unregistered handle {:?}", handle_id));
}

fn get_handle(handles: &HashMap<HandleId, WriterHandle>, id: &HandleId) -> Option<&WriterHandle> { handles.get(id) }

// avoid .expect/.unwrap on map lookups; use ok_or_else
let h = handles.remove(&id).ok_or_else(|| anyhow!("missing handle"))?;

When it happens

Trigger: Calling `alter_parallelisms` on a handle_id that was never registered via `NewHandle`, or calling `stop_handle` twice on the same HandleId (the first `remove` already deleted it). Any code path that removes handles outside this worker also breaks the invariant.

Common situations: Bugs in the sink coordination protocol state machine where a handle was stopped twice (e.g. a Stop event processed twice), a stale HandleId from a previous coordinator round being reused, or a refactored caller that forgets to add new handles before altering parallelisms.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• checked above
• infinite
• should have been handled
• IcebergSinkWriter should be initialized before barrier
• BatchPosixFsReader should not hit this branch. refer to `bat

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/7ae04b3a8adfca81. Report an issue: GitHub.