vectordotdev/vector · critical

Replacing unknown sink from fanout: {id}

Error message

Replacing unknown sink from fanout: {id}

What it means

Fanout is Vector's output bus: it fans one stream out to many sinks and is reconfigured at runtime through ControlMessages (Add/Remove/Pause/Replace) sent over its control channel during topology changes. replace() handles ControlMessage::Replace by looking up the sink's ComponentKey; if that key was never added (or was already removed), the lookup returns None and the fanout panics because the control-operation sequence is invalid.

Source

Thrown at lib/vector-core/src/fanout.rs:98

    fn remove(&mut self, id: &ComponentKey) {
        assert!(
            self.senders.shift_remove(id).is_some(),
            "Removing nonexistent sink from fanout: {id}"
        );
    }

    fn replace(&mut self, id: &ComponentKey, sink: BufferSender<EventArray>) {
        match self.senders.get_mut(id) {
            Some(sender) => {
                // While a sink must be _known_ to be replaced, it must also be empty (previously
                // paused or consumed when the `SendGroup` was created), otherwise an invalid
                // sequence of control operations has been applied.
                assert!(
                    sender.replace(Sender::new(sink)).is_none(),
                    "Replacing existing sink is not valid: {id}"
                );
            }
            None => panic!("Replacing unknown sink from fanout: {id}"),
        }
    }

    fn pause(&mut self, id: &ComponentKey) {
        match self.senders.get_mut(id) {
            Some(sender) => {
                // A sink must be known and present to be replaced, otherwise an invalid sequence of
                // control operations has been applied.
                assert!(
                    sender.take().is_some(),
                    "Pausing nonexistent sink is not valid: {id}"
                );
            }
            None => panic!("Pausing unknown sink from fanout: {id}"),
        }
    }

    /// Waits for the next control message and applies it.

View on GitHub (pinned to 3708c39b12)

Solutions

1. Upgrade Vector to the latest patch release and check the changelog for topology/reload fixes, since this panic indicates an internal control-sequence bug
2. Reproduce with a minimal config + reload sequence and capture RUST_LOG=trace logs of the ControlMessage order, then report it at github.com/vectordotdev/vector/issues
3. As a stopgap, restart Vector with the new config instead of live-reloading
4. If embedding vector-lib: only send Replace for an id previously Add-ed and Pause-d, and never after Remove

// Embedders driving the fanout control channel: only send Replace for an id
// that was previously Add-ed and Pause-d, and never after Remove.
control_tx.send(ControlMessage::Add(id.clone(), tx));     // 1. register
// ... later, during rebuild:
control_tx.send(ControlMessage::Pause(id.clone()));       // 2. pause
control_tx.send(ControlMessage::Replace(id.clone(), new_tx)); // 3. replace

When it happens

Trigger: A ControlMessage::Replace(id, sender) arriving for a ComponentKey that is not in the fanout's senders map: the id was never Add-ed, or a Remove(id) was processed before the Replace. This happens during topology rebuild (config reload / healthcheck-driven replacement) when the orchestrator and the fanout disagree on which sinks exist.

Common situations: Hitting 'vector reload' (SIGHUP or config file watch) with configs that add/remove sinks; historically this class of panic has been an internal Vector topology-rebuild bug fixed across releases; embedding code that drives the fanout ControlChannel by hand and sends Replace without a prior Add+Pause.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Pausing unknown sink from fanout: {id}
• join error or bad poll
• source output misconfigured - output for port {:?} missing
• errors ignored
• path and query should never fail to parse

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/e0ee6cbc2081edaf. Report an issue: GitHub.