{"record":{"id":"b3036d7ba8fbcab7","repo":"argoproj/argo-workflows","slug":"supervisor-presumed-dead-status-marker-never-appe","errorCode":null,"errorMessage":"supervisor presumed dead: status marker never appeared within %s","messagePattern":"supervisor presumed dead: status marker never appeared within (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":522,"sourceCode":"\t\t\treturn err\n\t\tcase <-ticker.C:\n\t\t\tif done, err := evaluateSupervisorStatus(statusPath, timeout, start); done {\n\t\t\t\tfinish(err)\n\t\t\t}\n\t\t}\n\t}\n}\n\n// evaluateSupervisorStatus reads the status marker once and decides whether main\n// can stop waiting. done=false means keep waiting. start is main's wait-start\n// reference, used to bound the case where the marker never appears at all. It is\n// safe to call concurrently — it only reads the filesystem.\nfunc evaluateSupervisorStatus(statusPath string, timeout time.Duration, start time.Time) (done bool, err error) {\n\tfi, statErr := os.Stat(statusPath)\n\tif statErr != nil {\n\t\tif os.IsNotExist(statErr) {\n\t\t\tif time.Since(start) > timeout {\n\t\t\t\treturn true, fmt.Errorf(\"supervisor presumed dead: status marker never appeared within %s\", timeout)\n\t\t\t}\n\t\t\treturn false, nil\n\t\t}\n\t\treturn true, fmt.Errorf(\"stat supervisor status: %w\", statErr)\n\t}\n\tbody, readErr := os.ReadFile(statusPath)\n\tif readErr != nil {\n\t\t// Stat just succeeded, so a read failure here means we raced the\n\t\t// supervisor's atomic rename (the old inode vanished between stat and\n\t\t// read). Treat it as transient and re-evaluate on the next tick/event\n\t\t// rather than failing the wait.\n\t\t//nolint:nilerr // deliberate: swallow the transient read error and retry\n\t\treturn false, nil\n\t}\n\ttoken, message := parseSupervisorStatus(body)\n\tswitch token {\n\tcase statusReady:\n\t\treturn true, nil","sourceCodeStart":504,"sourceCodeEnd":540,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L504-L540","documentation":"While waiting for the init-less supervisor to become ready, argoexec polls/watch the status marker at /var/run/argo/ctr/<name>/status. If the marker file never appears within supervisorHeartbeatTimeout of start, the supervisor is presumed dead and this error fails the wait (and the pod) instead of hanging until the pod deadline.","triggerScenarios":"evaluateSupervisorStatus sees os.Stat(statusPath) return IsNotExist continuously past the timeout — the supervisor binary never started or never wrote its first status marker.","commonSituations":"argoexec supervisor crashing at startup (bad image, missing template file); supervisor heartbeat goroutine dead; extremely slow node/kubelet starving the supervisor past the timeout; misbuilt image where the supervisor entrypoint is broken.","solutions":["Inspect argoexec/supervisor logs in the pod (kubectl logs -c <init/main container>) for a supervisor crash before the marker write","Verify the workflow image and argoexec version match (version skew between controller and executor can break the init-less beta protocol)","Disable the init-less beta feature (revert to legacy init/wait pod layout) via the appropriate executor/feature flag to unblock","Check node resource pressure (CPU throttling, OOM) that could prevent the supervisor from starting within the timeout"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# Before submit, confirm executor/controller version parity:\nargo version --short\nkubectl -n argo get deploy workflow-controller -o jsonpath='{.spec.template.spec.containers[0].image}'\n# versions must match; init-less mode is beta and protocol-sensitive","typeGuard":null,"tryCatchPattern":"if err := waitForSupervisorReady(ctx); err != nil {\n    if strings.Contains(err.Error(), \"presumed dead\") {\n        logger.Error(ctx, \"supervisor never became ready; check supervisor logs and image skew\", err)\n    }\n    return err\n}","preventionTips":["Pin controller and executor images to the same version","Check supervisor logs (kubectl logs) at the first occurrence, not after retries","Avoid nodes under severe CPU/memory pressure for workflow pods","Disable the init-less beta feature flag if it is not required for your workload"],"tags":["kubernetes","argo-workflows","argoexec","supervisor","timeout"],"backgroundTag":"supervisor-not-ready","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}