hashicorp/nomad · error

%s %s: %s

Error message

%s %s: %s

What it means

After the container was started, StartTask inspects it (ContainerInspect) to obtain fresh state; this error is thrown when the inspect call fails. The driver force-removes the container and returns a recoverable error, so the allocation can be retried on another placement or later. It generally means the daemon could not answer the inspect, typically because the daemon connection dropped or the container exited and was reaped between start and inspect.

Source

Thrown at drivers/docker/driver.go:431

			_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})
			// Some sort of docker race bug, recreating the container usually works
			if errdefs.IsConflict(err) && startAttempts < 5 {
				startAttempts++
				d.logger.Debug("reattempting container create/start sequence", "attempt", startAttempts, "container_id", id)
				goto CREATE
			}
			return nil, nil, nstructs.WrapRecoverable(fmt.Sprintf("Failed to start container %s: %s", container.Container.ID, err), err)
		}

		// Inspect container to get all of the container metadata as much of the
		// metadata (eg networking) isn't populated until the container is started
		runningContainer, err := dockerClient.ContainerInspect(d.ctx, container.Container.ID, mclient.ContainerInspectOptions{})
		if err != nil {
			_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})
			msg := "failed to inspect started container"
			d.logger.Error(msg, "error", err)
			_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})
			return nil, nil, nstructs.NewRecoverableError(fmt.Errorf("%s %s: %s", msg, container.Container.ID, err), true)
		}
		container = &runningContainer
		d.logger.Info("started container", "container_id", container.Container.ID)
	} else {
		d.logger.Debug("re-attaching to container", "container_id",
			container.Container.ID, "container_state", container.Container.State.Status)
	}

	collectingLogs := loggingIsEnabled(d.config, cfg)

	var dlogger docklog.DockerLogger
	var pluginClient *plugin.Client

	if collectingLogs {
		dlogger, pluginClient, err = d.setupNewDockerLogger(*container, cfg, time.Unix(0, 0))
		if err != nil {
			d.logger.Error("an error occurred after container startup, terminating container", "container_id", container.Container.ID)
			_, _ = dockerClient.ContainerRemove(d.ctx, container.Container.ID, mclient.ContainerRemoveOptions{Force: true})

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Check `docker logs`/daemon logs for why the container vanished immediately (fast-exiting entrypoint)
2. Verify the Docker daemon is stable and reachable (docker info); fix daemon crashes before retrying
3. Let the orchestrator retry - the error is marked recoverable, so rescheduling may succeed
4. If over TCP, check network stability/timeouts between agent and the remote Docker host

Example fix

// before
CMD ["myapp"]   # crashes instantly, container gone before inspect
// after
CMD ["myapp"] 
# plus verify locally: docker run --rm image myapp - stays up; fix app crash or missing env/config

// Verify the daemon is stable and the image entrypoint stays up before submitting:
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if _, err := dockerCli.Info(ctx); err != nil {
    return fmt.Errorf("daemon unreachable, task start will likely fail inspect: %w", err)
}
out, err := dockerCli.ContainerCreate(ctx, cfg, nil, nil, nil, "smoke-test")
_ = out
if err != nil { return err }

func isRecoverableInspectErr(err error) bool {
    return strings.Contains(err.Error(), "failed to inspect started container")
}

h, err := driver.StartTask(ctx, cfg)
if err != nil && strings.Contains(err.Error(), "failed to inspect started container") {
    // driver already force-removed the container and marked this recoverable
    time.Sleep(backoff)
    h, err = driver.StartTask(ctx, cfg) // safe to retry placement
}

When it happens

Trigger: dockerClient.ContainerInspect on the just-started container returns an error: daemon connection reset/timeout, container no longer exists (exited instantly and removed), or daemon restart mid-startup.

Common situations: Docker daemon crashed or restarted right after container start; container's entrypoint fails immediately so the container disappears before inspect; network interruption between agent and remote Docker daemon (DOCKER_HOST over tcp); long daemon stalls exceeding the client timeout.

Related errors

• Failed to pull `%s`: %w
• Failed to inspect container %s: %s
• failed to inspect container state: %v
• failed to inspect container %q: %v
• error creating bootstrap configuration for Connect proxy sid

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/a4a775745bc66a66. Report an issue: GitHub.