{"record":{"id":"a1efc7f76065f94c","repo":"argoproj/argo-workflows","slug":"supervisor-presumed-dead-no-status-update-within","errorCode":null,"errorMessage":"supervisor presumed dead: no status update within %s","messagePattern":"supervisor presumed dead: no status update within (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":547,"sourceCode":"\tif readErr != nil {\n\t\t// Stat just succeeded, so a read failure here means we raced the\n\t\t// supervisor's atomic rename (the old inode vanished between stat and\n\t\t// read). Treat it as transient and re-evaluate on the next tick/event\n\t\t// rather than failing the wait.\n\t\t//nolint:nilerr // deliberate: swallow the transient read error and retry\n\t\treturn false, nil\n\t}\n\ttoken, message := parseSupervisorStatus(body)\n\tswitch token {\n\tcase statusReady:\n\t\treturn true, nil\n\tcase statusFailed:\n\t\treturn true, fmt.Errorf(\"supervisor reported pre-main failure: %s\", message)\n\tdefault:\n\t\t// RUNNING, or a transient/partial read: the supervisor is alive only if\n\t\t// it is still heartbeating, i.e. the marker's mtime is fresh.\n\t\tif time.Since(fi.ModTime()) > timeout {\n\t\t\treturn true, fmt.Errorf(\"supervisor presumed dead: no status update within %s\", timeout)\n\t\t}\n\t\treturn false, nil\n\t}\n}\n\n// parseSupervisorStatus splits the marker into its first-line token and the\n// remaining message (used by the FAILED token to carry the cause).\nfunc parseSupervisorStatus(body []byte) (token, message string) {\n\tfirst, rest, _ := strings.Cut(string(body), \"\\n\")\n\treturn strings.TrimSpace(first), strings.TrimSpace(rest)\n}\n\n// waitForDependencyExitCode blocks until the given dependency exitcode file is\n// written, or until a SIGTERM/SIGKILL signal is received. It uses inotify on\n// the parent directory rather than polling.\n//\n// We deliberately do not select on ctx.Done() here: argoexec's root context is\n// bound to SIGTERM via signal.NotifyContext in main.go, so when SIGTERM","sourceCodeStart":529,"sourceCodeEnd":565,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L529-L565","documentation":"If the supervisor status marker exists but is stuck in a non-terminal state (RUNNING or a partial read) and its mtime has not advanced within the heartbeat timeout, argoexec concludes the supervisor stopped heartbeating and is presumed dead. The wait fails with this error rather than hanging until the pod deadline.","triggerScenarios":"evaluateSupervisorStatus reads a marker whose token is RUNNING (or unrecognized/partial) and time.Since(fi.ModTime()) exceeds supervisorHeartbeatTimeout — heartbeats stopped.","commonSituations":"Supervisor goroutine deadlocked or OOM-killed mid-run; node under heavy CPU pressure stalling heartbeats; supervisor crashed while main is still running; heartbeat interval misconfigured relative to a very short timeout.","solutions":["Check supervisor logs and pod events for OOM kills or crashes around the failure time","Verify controller/executor version compatibility — heartbeat protocol mismatches can silence updates","Check node load/kubelet eviction events; reschedule onto a less loaded node","Re-run the workflow; if reproducible, capture the marker contents (if any) and file an issue with argoexec logs"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# Confirm supervisor heartbeat health on similar nodes:\nkubectl top nodes   # avoid saturated nodes\nkubectl get events --field-selector type=Warning | grep -i oom  # prior OOM kills\n# Executor and controller versions must match:\nargo version --short","typeGuard":null,"tryCatchPattern":"if err := waitForSupervisorReady(ctx); err != nil {\n    if strings.Contains(err.Error(), \"no status update within\") {\n        log.Printf(\"supervisor heartbeats stopped; check for OOM/crash and node pressure\")\n    }\n    return err\n}","preventionTips":["Give workflow pods adequate memory limits to avoid supervisor OOM kills","Schedule on nodes without heavy CPU throttling","Keep controller/executor versions in lockstep","If reproducible, capture marker contents and argoexec logs for an upstream issue"],"tags":["kubernetes","argo-workflows","argoexec","supervisor","heartbeat-timeout"],"backgroundTag":"supervisor-not-ready","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}