hashicorp/nomad · warning

Failed to inspect container %s: %s

Error message

Failed to inspect container %s: %s

What it means

After listing containers, containerByName inspects the matched container to get its full state. If ContainerInspect fails — most often because the container was removed between the list and the inspect (a race) — the error is wrapped in a RecoverableError(true) so Nomad retries.

Source

Thrown at drivers/docker/driver.go:1722

OUTER:
	for _, shimContainer = range containers.Items {
		d.logger.Trace("listed container", "names", hclog.Fmt("%+v", shimContainer.Names))
		for _, name := range shimContainer.Names {
			if name == containerName {
				d.logger.Trace("Found container",
					"container_name", containerName, "container_id", shimContainer.ID)
				found = true
				break OUTER
			}
		}
	}
	if !found {
		return nil, nil
	}

	container, err := dockerClient.ContainerInspect(d.ctx, shimContainer.ID, mclient.ContainerInspectOptions{})
	if err != nil {
		err = fmt.Errorf("Failed to inspect container %s: %s", shimContainer.ID, err)

		// This error is always recoverable as it could
		// be caused by races between listing
		// containers and this container being removed.
		// See #2802
		return nil, nstructs.NewRecoverableError(err, true)
	}
	return &container, nil
}

// validateCommand validates that the command only has a single value and
// returns a user friendly error message telling them to use the passed
// argField.
func validateCommand(command, argField string) error {
	trimmed := strings.TrimSpace(command)
	if len(trimmed) == 0 {
		return fmt.Errorf("command empty: %q", command)
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Nothing to fix in the job — Nomad treats this as recoverable and retries; let the retry succeed.
2. Check for external tooling deleting containers (watchtower, cron docker rm) and exclude Nomad containers.
3. Verify no concurrent Docker daemon cleanup/GC is racing with Nomad on the host.

if err != nil {
    if nstructs.IsRecoverable(err) {
        time.Sleep(500 * time.Millisecond)
        return containerByName(name) // safe to retry: list-vs-inspect race
    }
    return nil, err
}

When it happens

Trigger: A race between ContainerList and ContainerInspect where the container is garbage-collected or stopped out of band (see Nomad #2802); or transient Docker API failures.

Common situations: Host under heavy container churn, out-of-band `docker rm`, Docker daemon GC, or external tooling (watchtower, kubelet-on-same-host) removing containers.

Related errors

• error creating bootstrap configuration for Connect proxy sid
• running container as ContainerAdmin is unsafe; change the co
• error decoding stats data: no reader body
• error decoding stats data: stats were nil
• does not match registry specification

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/86d2f7e3fdbdebc1. Report an issue: GitHub.