hashicorp/nomad · error

failed to inspect container for id %q: %v

Error message

failed to inspect container for id %q: %v

What it means

RecoverTask inspects the persisted container (handleState.ContainerID) to rebuild the task handle. If the Docker API inspect call fails — most commonly because the container no longer exists — recovery of that task is impossible and this error names the offending container ID.

Source

Thrown at drivers/docker/driver.go:270

	dockerClient, err := d.getDockerClient()
	if err != nil {
		return fmt.Errorf("failed to get docker client: %w", err)
	}

	dockerInfo, err := dockerClient.Info(d.ctx, mclient.InfoOptions{})
	if err != nil {
		return fmt.Errorf("failed to fetch docker daemon info: %v", err)
	}

	infinityClient, err := d.getInfinityClient()
	if err != nil {
		return fmt.Errorf("failed to get docker long operations client: %w", err)
	}

	container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})
	if err != nil {
		return fmt.Errorf("failed to inspect container for id %q: %v", handleState.ContainerID, err)
	}

	h := &taskHandle{
		dockerClient:            dockerClient,
		dockerCGroupDriver:      dockerInfo.Info.CgroupDriver,
		infinityClient:          infinityClient,
		logger:                  d.logger.With("container_id", container.Container.ID),
		task:                    handle.Config,
		containerID:             container.Container.ID,
		containerCgroup:         string(container.Container.HostConfig.Cgroup),
		containerImage:          container.Container.Image,
		doneCh:                  make(chan bool),
		waitCh:                  make(chan struct{}),
		removeContainerOnExit:   d.config.GC.Container,
		net:                     handleState.DriverNetwork,
		disableCpusetManagement: d.config.disableCpusetManagement,
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check whether the container exists: `docker inspect <containerID>`. If it is gone, the task cannot be recovered — kill/forget the task handle instead of retrying.
2. Verify DOCKER_HOST/endpoint points to the same daemon that originally ran the container.
3. If containers were GC'd, disable aggressive container cleanup (d.config.GC.Container) or cleanup scripts on hosts running recoverable tasks.
4. If the ID is stale in the state store, clear the corrupted task state so the task can be rescheduled fresh.

Example fix

// before
container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})
if err != nil {
	return fmt.Errorf("failed to inspect container for id %q: %v", handleState.ContainerID, err)
}
// after
container, err := dockerClient.ContainerInspect(d.ctx, handleState.ContainerID, mclient.ContainerInspectOptions{})
if err != nil {
	if mclient.IsErrNotFound(err) {
		d.logger.Warn("container gone; cannot recover task", "container_id", handleState.ContainerID)
		return drivers.ErrTaskNotFound // signal unrecoverable instead of generic error
	}
	return fmt.Errorf("failed to inspect container for id %q: %w", handleState.ContainerID, err)
}

// check container existence before attempting recovery
func containerExists(ctx context.Context, cli *client.Client, id string) (bool, error) {
	_, err := cli.ContainerInspect(ctx, id)
	if err != nil {
		if errdefs.IsNotFound(err) {
			return false, nil
		}
		return false, err
	}
	return true, nil
}

func isContainerNotFound(err error) bool {
	return errdefs.IsNotFound(err) // or strings.Contains(err.Error(), "No such container")
}

// Go: treat not-found as unrecoverable, other errors as retryable
if err := driver.RecoverTask(handle); err != nil {
	if strings.Contains(err.Error(), "failed to inspect container") {
		if isContainerNotFound(errors.Unwrap(err)) {
			// drop the handle / reschedule the task; do not retry
		} else {
			// transient daemon error: retry with backoff
		}
	}
}

When it happens

Trigger: Container was removed manually (`docker rm`) or by a GC/cleanup script between the client crash and recovery; container ID in the persisted state is stale or corrupted; daemon was restarted with a different data root; inspect times out or the daemon errors.

Common situations: Operator cleaned up 'orphaned' containers while the orchestrator was down; Docker upgraded/reinstalled losing containers; state file references a container from a different daemon (DOCKER_HOST changed); disk pressure cleanup daemon deleting containers.

Related errors

• failed to fetch docker daemon info: %v
• failed to get docker long operations client: %w
• failed to setup replacement docker logger: %v
• failed to store driver state: %v
• running container as ContainerAdmin is unsafe; change the co

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/e21b20fefd934d93. Report an issue: GitHub.