{"record":{"id":"c9d39769bfd1db77","repo":"hashicorp/nomad","slug":"oom-killed","errorCode":null,"errorMessage":"OOM Killed","messagePattern":"OOM Killed","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/handle.go","lineNumber":324,"sourceCode":"\tdefer inspectCancel()\n\n\tcontainer, ierr := h.dockerClient.ContainerInspect(ctx, h.containerID, mclient.ContainerInspectOptions{})\n\toom := false\n\tif ierr != nil {\n\t\th.logger.Error(\"failed to inspect container\", \"error\", ierr)\n\t} else if container.Container.State.OOMKilled {\n\t\th.logger.Error(\"OOM Killed\",\n\t\t\t\"container_id\", h.containerID,\n\t\t\t\"container_image\", h.containerImage,\n\t\t\t\"nomad_job_name\", h.task.JobName,\n\t\t\t\"nomad_task_name\", h.task.Name,\n\t\t\t\"nomad_alloc_id\", h.task.AllocID)\n\n\t\t// Note that with cgroups.v2 the cgroup OOM killer is not\n\t\t// observed by docker container status. But we can't test the\n\t\t// exit code, as 137 is used for any SIGKILL\n\t\toom = true\n\t\twerr = fmt.Errorf(\"OOM Killed\")\n\t}\n\n\t// Shutdown stats collection\n\tclose(h.doneCh)\n\n\t// Stop the container just incase the docker daemon's wait returned\n\t// incorrectly. Container should have exited by now so kill_timeout can be\n\t// ignored.\n\tctx, stopCancel := context.WithTimeout(context.Background(), 10*time.Second)\n\tdefer stopCancel()\n\tif _, err := h.dockerClient.ContainerStop(ctx, h.containerID, mclient.ContainerStopOptions{\n\t\tTimeout: new(0),\n\t}); err != nil {\n\t\tif !errdefs.IsNotModified(err) && !errdefs.IsNotFound(err) {\n\t\t\th.logger.Error(\"error stopping container\", \"error\", err)\n\t\t}\n\t}\n","sourceCodeStart":306,"sourceCodeEnd":342,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/handle.go#L306-L342","documentation":"This error is produced by the Docker driver's task run loop when, after the container exits, a ContainerInspect shows State.OOMKilled=true. It replaces the normal exit-code error because 137 is ambiguous (any SIGKILL), so the driver explicitly reports the container was killed by the kernel's out-of-memory killer.","triggerScenarios":"A task's Docker container exits and the post-exit ContainerInspect reports OOMKilled=true; raised in run(), which RecoverTask and StartTask call. Happens when the process exceeds its memory hard limit under cgroups v1 (the cgroup OOM kill is not observed by Docker status under cgroups v2).","commonSituations":"Tasks with undersized resources.memory; memory leaks in the job; host memory pressure causing cgroup OOM kills; workloads with bursty allocation spikes exceeding the memory limit.","solutions":["Increase the task's resources.memory in the job spec so the container fits its working set","Profile the application for memory leaks or reduce its memory footprint","Check host memory pressure and other allocations competing for RAM","For cgroups v2 hosts, monitor memory events rather than relying on Docker OOMKilled status, as the kill may not be surfaced"],"exampleFix":"// before\ntask \"worker\" {\n  driver = \"docker\"\n  resources { memory = 128 }\n}\n// after\ntask \"worker\" {\n  driver = \"docker\"\n  resources { memory = 512 }\n}","handlingStrategy":"validation","validationCode":"// check memory limits before deploying\nif task.Resources.MemoryMB < appProfile.MinMemoryMB {\n    return fmt.Errorf(\"task %s: %d MB is below observed working set of %d MB\",\n        task.Name, task.Resources.MemoryMB, appProfile.MinMemoryMB)\n}","typeGuard":"func isOOMKilled(err error) bool { return err != nil && strings.Contains(err.Error(), \"OOM Killed\") }","tryCatchPattern":"if isOOMKilled(err) {\n    log.Warn(\"container OOM killed; rescheduling with raised memory\")\n    rescheduleWithBumpedMemory(task)\n} else {\n    return err\n}","preventionTips":["Set resources.memory from measured peak RSS, not guesses","Enable memory swelling headroom (10-20%) for bursty workloads","Alert on cgroup memory events / oom_kill counters","Watch for cgroups v2 hosts where OOMKilled may not be reported"],"tags":["docker","oom","memory-limit","container"],"backgroundTag":"container-oom-killed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}