hashicorp/nomad · error

OOM Killed

Error message

OOM Killed

What it means

This error is produced by the Docker driver's task run loop when, after the container exits, a ContainerInspect shows State.OOMKilled=true. It replaces the normal exit-code error because 137 is ambiguous (any SIGKILL), so the driver explicitly reports the container was killed by the kernel's out-of-memory killer.

Source

Thrown at drivers/docker/handle.go:324

	defer inspectCancel()

	container, ierr := h.dockerClient.ContainerInspect(ctx, h.containerID, mclient.ContainerInspectOptions{})
	oom := false
	if ierr != nil {
		h.logger.Error("failed to inspect container", "error", ierr)
	} else if container.Container.State.OOMKilled {
		h.logger.Error("OOM Killed",
			"container_id", h.containerID,
			"container_image", h.containerImage,
			"nomad_job_name", h.task.JobName,
			"nomad_task_name", h.task.Name,
			"nomad_alloc_id", h.task.AllocID)

		// Note that with cgroups.v2 the cgroup OOM killer is not
		// observed by docker container status. But we can't test the
		// exit code, as 137 is used for any SIGKILL
		oom = true
		werr = fmt.Errorf("OOM Killed")
	}

	// Shutdown stats collection
	close(h.doneCh)

	// Stop the container just incase the docker daemon's wait returned
	// incorrectly. Container should have exited by now so kill_timeout can be
	// ignored.
	ctx, stopCancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer stopCancel()
	if _, err := h.dockerClient.ContainerStop(ctx, h.containerID, mclient.ContainerStopOptions{
		Timeout: new(0),
	}); err != nil {
		if !errdefs.IsNotModified(err) && !errdefs.IsNotFound(err) {
			h.logger.Error("error stopping container", "error", err)
		}
	}

View on GitHub (pinned to 482b49bf1a)

Solutions

1. Increase the task's resources.memory in the job spec so the container fits its working set
2. Profile the application for memory leaks or reduce its memory footprint
3. Check host memory pressure and other allocations competing for RAM
4. For cgroups v2 hosts, monitor memory events rather than relying on Docker OOMKilled status, as the kill may not be surfaced

Example fix

// before
task "worker" {
  driver = "docker"
  resources { memory = 128 }
}
// after
task "worker" {
  driver = "docker"
  resources { memory = 512 }
}

// check memory limits before deploying
if task.Resources.MemoryMB < appProfile.MinMemoryMB {
    return fmt.Errorf("task %s: %d MB is below observed working set of %d MB",
        task.Name, task.Resources.MemoryMB, appProfile.MinMemoryMB)
}

func isOOMKilled(err error) bool { return err != nil && strings.Contains(err.Error(), "OOM Killed") }

if isOOMKilled(err) {
    log.Warn("container OOM killed; rescheduling with raised memory")
    rescheduleWithBumpedMemory(task)
} else {
    return err
}

When it happens

Trigger: A task's Docker container exits and the post-exit ContainerInspect reports OOMKilled=true; raised in run(), which RecoverTask and StartTask call. Happens when the process exceeds its memory hard limit under cgroups v1 (the cgroup OOM kill is not observed by Docker status under cgroups v2).

Common situations: Tasks with undersized resources.memory; memory leaks in the job; host memory pressure causing cgroup OOM kills; workloads with bursty allocation spikes exceeding the memory limit.

Related errors

• failed to launch docker logger process %s: %v
• running container as ContainerAdmin is unsafe; change the co
• error decoding stats data: no reader body
• error decoding stats data: stats were nil
• does not match registry specification

AI-assisted analysis of hashicorp/nomad@482b49bf1a (2026-09-04). Data as JSON: /api/errors/c9d39769bfd1db77. Report an issue: GitHub.