hashicorp/nomad · error

failed to inspect container %q: %v

Error message

failed to inspect container %q: %v

What it means

This error wraps a failure from the Docker Engine API `ContainerInspect` call while building a task's status snapshot in the Nomad Docker driver. The driver inspects the container to read State.StartedAt/FinishedAt and derive task status; if the Docker daemon cannot return the container metadata, the underlying error is wrapped with the container ID for context. It means the driver has a handle for the task but the container backing it is no longer visible or the API call failed.

Source

Thrown at drivers/docker/driver.go:1853

	d.coordinator.RemoveImage(handle.containerImage, handle.task.ID)

	return nil
}

func (d *Driver) InspectTask(taskID string) (*drivers.TaskStatus, error) {
	h, ok := d.tasks.Get(taskID)
	if !ok {
		return nil, drivers.ErrTaskNotFound
	}

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return nil, err
	}

	container, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})
	if err != nil {
		return nil, fmt.Errorf("failed to inspect container %q: %v", h.containerID, err)
	}

	started, _ := time.Parse(time.RFC3339, container.Container.State.StartedAt)
	completed, _ := time.Parse(time.RFC3339, container.Container.State.FinishedAt)

	status := &drivers.TaskStatus{
		ID:          h.task.ID,
		Name:        h.task.Name,
		StartedAt:   started,
		CompletedAt: completed,
		DriverAttributes: map[string]string{
			"container_id": container.Container.ID,
		},
		NetworkOverride: h.net,
		ExitResult:      h.ExitResult(),
	}

	status.State = drivers.TaskStateUnknown

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check the container still exists with `docker ps -a | grep <containerID>`; if removed, re-run the task or clear the stale handle.
2. Verify the Docker daemon is healthy: `docker info` and `systemctl status docker`.
3. Inspect client/daemon connectivity (DOCKER_HOST, socket permissions, remote API reachability).
4. Upgrade the Docker client library/daemon if ContainerInspect intermittently fails after daemon restarts.

Example fix

// before
container, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})
if err != nil {
    return nil, fmt.Errorf("failed to inspect container %q: %v", h.containerID, err)
}
// after
ccontainer, err := dockerClient.ContainerInspect(d.ctx, h.containerID, mclient.ContainerInspectOptions{})
if err != nil {
    if errdefs.IsNotFound(err) {
        return nil, fmt.Errorf("container %q no longer exists: %w", h.containerID, err)
    }
    return nil, fmt.Errorf("failed to inspect container %q: %v", h.containerID, err)
}

func isContainerNotFound(err error) bool { return errdefs.IsNotFound(err) || strings.Contains(err.Error(), "No such container") }

status, err := driver.InspectTask(taskID)
if err != nil {
    if isContainerNotFound(err) {
        // treat task as dead; cleanup handle
        return drivers.ErrTaskNotFound
    }
    return fmt.Errorf("status inspection failed, check dockerd health: %w", err)
}

When it happens

Trigger: Calling the driver's InsTaskStatus path when dockerClient.ContainerInspect(ctx, h.containerID, ...) returns an error — typically because the container was removed concurrently, the container ID is stale, or the Docker daemon is unreachable.

Common situations: Container exited and was auto-removed (or garbage collected) before status was inspected; Docker daemon restart/OOM; task handle outlived the container after a node event; network interruption between Nomad and dockerd.

Related errors

• Failed to create long operations docker client: %v
• %s %s: %s
• failed to inspect container state: %v
• DriverStatsNotImplemented
• running container as ContainerAdmin is unsafe; change the co

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/794e48feb73dfc5f. Report an issue: GitHub.