{"record":{"id":"b956d2b04cd62491","repo":"hashicorp/nomad","slug":"failed-to-stop-allocation-w","errorCode":null,"errorMessage":"Failed to stop allocation: %w","messagePattern":"Failed to stop allocation: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"command/job_restart.go","lineNumber":955,"sourceCode":"\t\tformatTime(time.Now()),\n\t\tshortAllocID,\n\t\talloc.TaskGroup,\n\t))\n\n\tvar q *api.QueryOptions\n\n\tif c.noShutdownDelay {\n\t\tq = &api.QueryOptions{\n\t\t\tParams: map[string]string{\"no_shutdown_delay\": \"true\"},\n\t\t}\n\t}\n\n\t// Stop allocation and wait for its replacement to be running or for a\n\t// blocked evaluation that prevents placements for this task group to\n\t// happen.\n\tresp, err := c.client.Allocations().Stop(&api.Allocation{ID: alloc.ID}, q)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"Failed to stop allocation: %w\", err)\n\t}\n\n\t// Allocations for system jobs do not get replaced by the scheduler after\n\t// being stopped, so an eval is needed to trigger the reconciler.\n\tif alloc.isSystemJob() {\n\t\topts := api.EvalOptions{\n\t\t\tForceReschedule: true,\n\t\t}\n\t\t_, _, err := c.client.Jobs().EvaluateWithOpts(*alloc.Job.ID, opts, nil)\n\t\tif err != nil {\n\t\t\treturn fmt.Errorf(\"Failed evaluate job: %w\", err)\n\t\t}\n\t}\n\n\t// errCh receives an error if anything goes wrong or nil when the\n\t// replacement allocation is running.\n\t// Use a buffered channel to prevent both goroutine from blocking trying to\n\t// send a result back.","sourceCodeStart":937,"sourceCodeEnd":973,"githubUrl":"https://github.com/hashicorp/nomad/blob/482b49bf1aec006f089bcfc7e632d8f6ac303e5e/command/job_restart.go#L937-L973","documentation":"Fires in stopAlloc during job restart: the Allocations().Stop API call to the server failed, so the allocation was not stopped and no replacement will be scheduled for it.","triggerScenarios":"The Stop RPC fails — agent unreachable, alloc already garbage-collected/terminal, context/query timeout expired, or ACL token lacks permission.","commonSituations":"Alloc was replaced by the scheduler between listing and stopping (stale ID); node down so the client RPC fails; too-short -detach or query timeout during a slow drain; missing alloc-lifecycle ACL capability.","solutions":["Re-run nomad job restart; the command will pick up the current alloc IDs","Check nomad alloc status and agent health/network connectivity","Extend the query timeout / wait options used by the command","Ensure the ACL token has permissions to stop allocations"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"alloc, _, err := client.Allocations().Info(allocID, nil)\nif err != nil {\n    return err\n}\nif alloc.DesiredStatus == \"stop\" || isTerminal(alloc.ClientStatus) {\n    return fmt.Errorf(\"alloc %s already stopping/terminal\", allocID)\n}","typeGuard":"func isStoppable(a *api.Allocation) bool {\n    return a != nil && a.DesiredStatus != \"stop\" && a.ClientStatus == \"running\"\n}","tryCatchPattern":"_, _, err := client.Allocations().Stop(&api.Allocation{ID: allocID}, q)\nif err != nil {\n    if errors.Is(err, context.DeadlineExceeded) || isTransientRPC(err) {\n        return retryWithBackoff(func() error { _, _, err := client.Allocations().Stop(...); return err })\n    }\n    return err\n}","preventionTips":["Re-fetch the alloc before stopping to avoid stale IDs","Use adequate query timeouts/waits","Ensure ACL token has alloc-lifecycle capability"],"tags":["nomad","api","allocation","stop","rpc"],"backgroundTag":"alloc-stop-failed","analyzedSha":"482b49bf1aec006f089bcfc7e632d8f6ac303e5e","analyzedAt":"2026-09-04T07:54:14.808Z","contentChangedAt":"2026-09-04T07:54:14.808Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}