vectordotdev/vector · error

Every started ContainerId has it's ContainerState

Error message

Every started ContainerId has it's ContainerState

What it means

In the docker_logs actor's error branch, a failed log stream reports (id, ErrorPersistence); the actor removes its state with containers.remove(&id).expect("Every started ContainerId has it's ContainerState"). The invariant: an error can only arrive for a container the actor started and still tracks. The panic fires when an error arrives for an id that was already removed or never inserted - the same bookkeeping desync as the info path, seen from the failure side.

Solutions

1. Gather logs (Vector version, docker events, debug output) and report upstream as an internal invariant break
2. Upgrade/downgrade to a docker_logs-stable release
3. Patch: use containers.remove(&id) and warn-and-continue on None instead of expecting
4. Supervise and restart Vector on this panic until a fixed build is deployed

Example fix

// before
let state = self
    .containers
    .remove(&id)
    .expect("Every started ContainerId has it's ContainerState");

// after
let Some(state) = self.containers.remove(&id) else {
    warn!(message = "error reported for unknown container, skipping", id = %id);
    continue;
};

// before removing in the error branch:
if let Some(state) = self.containers.remove(&id) {
    // existing ErrorPersistence handling
} else {
    warn!(message = "error for untracked container", id = %id);
    continue;
}

fn remove_state(
    containers: &mut HashMap<ContainerId, ContainerState>,
    id: &ContainerId,
) -> Option<ContainerState> {
    containers.remove(id)
}

match self.containers.remove(&id) {
    Some(state) => match persistence {
        ErrorPersistence::Transient => { /* restart */ }
        ErrorPersistence::Permanent => (),
    },
    None => warn!(id = %id, "duplicate or unknown container error"),
}

When it happens

Trigger: An error message for a container whose state was already removed (double-report of a failed stream, or a race between return_info and the error report for the same id).

Common situations: Docker daemon restart storms, aggressive container churn, or docker_logs refactors/regressions in a given Vector version.

Related errors

• Every ContainerLogInfo has it's ContainerState
• `docker_logs.stream` must exist in the metadata
• `message` must exist in the event
• `.` must exist in the event
• stream must exist in the event

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/c65a6d965df20c04. Report an issue: GitHub.

Appendix: source

Thrown at src/sources/docker_logs/mod.rs:606

    async fn run(mut self) {
        loop {
            tokio::select! {
                value = self.main_recv.recv() => {
                    match value {
                        Some(Ok(info)) => {
                            let state = self
                                .containers
                                .get_mut(&info.id)
                                .expect("Every ContainerLogInfo has it's ContainerState");
                            if state.return_info(info) {
                                self.esb.restart(state);
                            }
                        },
                        Some(Err((id,persistence))) => {
                            let state = self
                                .containers
                                .remove(&id)
                                .expect("Every started ContainerId has it's ContainerState");
                            match persistence{
                                ErrorPersistence::Transient => if state.is_running() {
                                    let backoff= Some(self.backoff_duration);
                                    self.containers.insert(id.clone(), self.esb.start(id, backoff));
                                }
                                // Forget the container since the error is permanent.
                                ErrorPersistence::Permanent => (),
                            }
                        }
                        None => {
                            error!(message = "The docker_logs source main stream has ended unexpectedly.", internal_log_rate_limit = false);
                            info!(message = "Shutting down docker_logs source.");
                            return;
                        }
                    };
                }
                value = self.events.next() => {
                    match value {

View on GitHub (pinned to 3708c39b12)