hashicorp/nomad · error

failed to inspect container state: %v

Error message

failed to inspect container state: %v

What it means

DestroyTask inspects the container before removing it. If the inspect call fails with any error other than 'not found' (daemon down, timeout, permission), the destroy aborts with this error instead of proceeding, because Nomad cannot verify the container's state.

Source

Thrown at drivers/docker/driver.go:1797

func (d *Driver) DestroyTask(taskID string, force bool) error {
	h, ok := d.tasks.Get(taskID)
	if !ok {
		return drivers.ErrTaskNotFound
	}

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return err
	}

	c, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})
	if err != nil {
		if errdefs.IsNotFound(err) {
			h.logger.Info("container was removed out of band, will proceed with DestroyTask",
				"error", err)
		} else {
			return fmt.Errorf("failed to inspect container state: %v", err)
		}
	} else {
		if c.Container.State.Running {
			if !force {
				return fmt.Errorf("must call StopTask for the given task before Destroy or set force to true")
			}
			if _, err := dockerClient.ContainerStop(d.ctx, h.containerID, mclient.ContainerStopOptions{Timeout: new(0)}); err != nil {
				h.logger.Warn("failed to stop container during destroy", "error", err)
			}
		}

		if h.removeContainerOnExit {
			if _, err := dockerClient.ContainerRemove(d.ctx, h.containerID, mclient.ContainerRemoveOptions{Force: true}); err != nil {
				h.logger.Error("error removing container", "error", err)
			}
		} else {
			h.logger.Debug("not removing container due to config")
		}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Verify the Docker daemon is healthy (`docker ps`, `systemctl status docker`) and retry the destroy/GC.
2. If the container is already gone but inspect errored transiently, the next GC pass will remove the leftover tracking state.
3. Check docker.sock permissions and docker logs for API errors; restart the daemon if wedged.

Example fix

// before
sudo docker ps   # hangs or errors
// after
sudo systemctl restart docker
nomad node drain <node> -enable   # or let alloc GC retry

// preflight before destroy/GC: daemon must answer
cmd := exec.Command("docker", "info")
if err := cmd.Run(); err != nil { log.Printf("docker daemon down; defer destroy") }

if err := destroy(); err != nil {
    if strings.Contains(err.Error(), "failed to inspect container state") {
        time.Sleep(retryBackoff)
        return destroy() // transient daemon issue; retry after docker recovers
    }
    return err
}

When it happens

Trigger: Docker daemon unreachable or timing out during DestroyTask; transient API errors that are not errdefs.IsNotFound while tearing down an allocation.

Common situations: Docker daemon hung under load, socket permission changes, host under heavy I/O making the API time out during alloc GC or node drain.

Related errors

• Failed to query list of containers: %s
• running container as ContainerAdmin is unsafe; change the co
• error decoding stats data: no reader body
• error decoding stats data: stats were nil
• does not match registry specification

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/7b70b38724fd8a51. Report an issue: GitHub.