hashicorp/nomad · error

Docker container exited with non-zero exit code: %d

Error message

Docker container exited with non-zero exit code: %d

What it means

After a task's container exits, the driver waits on it (WaitConditionNotRunning) and, when the exit status code is non-zero, wraps it as 'Docker container exited with non-zero exit code: <code>'. This is the standard way the docker driver surfaces application failure — the container ran but the process returned a failing exit code.

Source

Thrown at drivers/docker/handle.go:299

func (h *taskHandle) run() {
	defer h.shutdownLogger()

	h.startCpusetFixer()

	var werr error
	var exitCode containerapi.WaitResponse
	// this needs to use the background context because the container can
	// outlive Nomad itself
	waitResult := h.infinityClient.ContainerWait(
		context.Background(),
		h.containerID,
		mclient.ContainerWaitOptions{Condition: containerapi.WaitConditionNotRunning},
	)

	select {
	case exitCode = <-waitResult.Result:
		if exitCode.StatusCode != 0 {
			werr = fmt.Errorf("Docker container exited with non-zero exit code: %d", exitCode.StatusCode)
		}
	case werr = <-waitResult.Error:
		h.logger.Error("failed to wait for container; already terminated")
	}

	ctx, inspectCancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer inspectCancel()

	container, ierr := h.dockerClient.ContainerInspect(ctx, h.containerID, mclient.ContainerInspectOptions{})
	oom := false
	if ierr != nil {
		h.logger.Error("failed to inspect container", "error", ierr)
	} else if container.Container.State.OOMKilled {
		h.logger.Error("OOM Killed",
			"container_id", h.containerID,
			"container_image", h.containerImage,
			"nomad_job_name", h.task.JobName,
			"nomad_task_name", h.task.Name,

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Read the task's stdout/stderr via 'nomad alloc logs <alloc>' to find the application-level failure.
2. Map the exit code: 137/143 = OOM or SIGKILL/SIGTERM (check memory resources), 1 = app error, 125/126 = container/command setup error.
3. Fix the app bug, command, or configuration in the job spec and redeploy.
4. Increase task memory/CPU if the exit was resource-related.

Example fix

// before: app crashes on missing env
env { MISSING_VAR = "" }
// after
env { MISSING_VAR = "required-value" }

// validate entrypoint/cmd args and required env before starting the task
for _, v := range requiredEnv {
    if os.Getenv(v) == "" {
        return fmt.Errorf("missing required env %s", v)
    }
}

ev, err := client.Allocations().Logs(ctx, alloc, task, "stdout", false, nil)
if err == nil {
    log.Printf("task output: %s", ev)
}
// inspect exit code to branch: 137/143 resource/kill vs app error

When it happens

Trigger: The containerized process terminated by itself with a non-zero status (the select on waitResult.Result yields StatusCode != 0). Not a Docker API error — the wait succeeded, the app failed.

Common situations: Application crash, unhandled exception, failed health/startup checks, bad entrypoint arguments, OOM-kill (exit 137), config errors in the app; exit 125/126 for Docker-level command errors.

Related errors

• failed to inspect exit code of exec object: %w
• websocket closed before receiving exit code: %w
• Malformed hard ulimit %v: %v
• cannot apply %q configuration
• failed to create exec object: %v

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/58c8bf2093dcca85. Report an issue: GitHub.