{"record":{"id":"b7fd7636b118edcc","repo":"hashicorp/nomad","slug":"failed-to-signal-container-q-while-killing-v","errorCode":null,"errorMessage":"Failed to signal container %q while killing: %v","messagePattern":"Failed to signal container %q while killing: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"drivers/docker/handle.go","lineNumber":197,"sourceCode":"\t\t}\n\n\t\tctx, cancel := context.WithTimeout(context.Background(), killTimeout)\n\t\tdefer cancel()\n\n\t\tif err := h.Signal(ctx, signal); err != nil {\n\t\t\t// Container has already been removed.\n\t\t\tif errdefs.IsNotFound(err) {\n\t\t\t\th.logger.Debug(\"attempted to signal nonexistent container\")\n\t\t\t\treturn nil\n\t\t\t}\n\t\t\t// Container has already been stopped.\n\t\t\tif errdefs.IsNotModified(err) {\n\t\t\t\th.logger.Debug(\"attempted to signal a not-running container\")\n\t\t\t\treturn nil\n\t\t\t}\n\n\t\t\th.logger.Error(\"failed to signal container while killing\", \"error\", err)\n\t\t\treturn fmt.Errorf(\"Failed to signal container %q while killing: %v\", h.containerID, err)\n\t\t}\n\n\t\tselect {\n\t\tcase <-h.waitCh:\n\t\t\treturn nil\n\t\tcase <-ctx.Done():\n\t\t}\n\n\t\t// Stop the container forcefully.\n\t\t_, err = h.dockerClient.ContainerStop(context.Background(), h.containerID, mclient.ContainerStopOptions{Timeout: new(0)})\n\t}\n\n\tif err != nil {\n\t\t// Container has already been removed.\n\t\tif errdefs.IsNotFound(err) {\n\t\t\th.logger.Debug(\"attempted to stop nonexistent container\")\n\t\t\treturn nil\n\t\t}","sourceCodeStart":179,"sourceCodeEnd":215,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/drivers/docker/handle.go#L179-L215","documentation":"During Kill, after the initial signal the driver waits for the container to exit and, if needed, signals it again. If that signaling call fails with an error that is not errdefs.IsNotModified (i.e. the container is genuinely still running and rejecting the signal), Kill returns this wrapped error naming the container ID.","triggerScenarios":"h.dockerClient.ContainerKill (via h.Signal) during kill returns a real API error — daemon connection failure, no such container (removed externally), permission denied — and the error is not 'not modified'.","commonSituations":"Docker daemon restarted mid-kill; container removed by an external supervisor while Nomad was killing it; socket permission changes; engine unresponsive under load.","solutions":["Check Docker daemon health and logs on the host; restart if it is unresponsive.","Verify the container still exists ('docker ps -a | grep <id>') — external removal causes no-such-container errors.","Check Nomad client's access to the Docker socket.","Retry the alloc stop; if the container already died, the error will disappear."],"exampleFix":"// before: fighting an unhealthy daemon during kill\nnomad alloc stop <alloc>  # fails: Failed to signal container ...\n// after: heal the daemon first\nsudo systemctl restart docker && nomad alloc stop <alloc>","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"err := client.Nodes.StopAlloc(ctx, allocID, nil)\nif err != nil && strings.Contains(err.Error(), \"Failed to signal container\") {\n    // container may have died anyway; verify before escalating\n    st, serr := client.Allocations().Info(ctx, allocID, nil)\n    if serr == nil && st.TaskStates[task].Finished { return nil }\n    return err\n}","preventionTips":["Monitor Docker daemon health; kills fail when the engine is down.","Avoid external tooling that removes Nomad-managed containers.","Retry alloc stop once before investigating deeper."],"tags":["docker-driver","kill","signal","docker-api"],"backgroundTag":"container-kill-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}