argoproj/argo-workflows · error

supervisor presumed dead: status marker never appeared withi

Error message

supervisor presumed dead: status marker never appeared within %s

What it means

While waiting for the init-less supervisor to become ready, argoexec polls/watch the status marker at /var/run/argo/ctr/<name>/status. If the marker file never appears within supervisorHeartbeatTimeout of start, the supervisor is presumed dead and this error fails the wait (and the pod) instead of hanging until the pod deadline.

Source

Thrown at cmd/argoexec/commands/emissary.go:522

			return err
		case <-ticker.C:
			if done, err := evaluateSupervisorStatus(statusPath, timeout, start); done {
				finish(err)
			}
		}
	}
}

// evaluateSupervisorStatus reads the status marker once and decides whether main
// can stop waiting. done=false means keep waiting. start is main's wait-start
// reference, used to bound the case where the marker never appears at all. It is
// safe to call concurrently — it only reads the filesystem.
func evaluateSupervisorStatus(statusPath string, timeout time.Duration, start time.Time) (done bool, err error) {
	fi, statErr := os.Stat(statusPath)
	if statErr != nil {
		if os.IsNotExist(statErr) {
			if time.Since(start) > timeout {
				return true, fmt.Errorf("supervisor presumed dead: status marker never appeared within %s", timeout)
			}
			return false, nil
		}
		return true, fmt.Errorf("stat supervisor status: %w", statErr)
	}
	body, readErr := os.ReadFile(statusPath)
	if readErr != nil {
		// Stat just succeeded, so a read failure here means we raced the
		// supervisor's atomic rename (the old inode vanished between stat and
		// read). Treat it as transient and re-evaluate on the next tick/event
		// rather than failing the wait.
		//nolint:nilerr // deliberate: swallow the transient read error and retry
		return false, nil
	}
	token, message := parseSupervisorStatus(body)
	switch token {
	case statusReady:
		return true, nil

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect argoexec/supervisor logs in the pod (kubectl logs -c <init/main container>) for a supervisor crash before the marker write
2. Verify the workflow image and argoexec version match (version skew between controller and executor can break the init-less beta protocol)
3. Disable the init-less beta feature (revert to legacy init/wait pod layout) via the appropriate executor/feature flag to unblock
4. Check node resource pressure (CPU throttling, OOM) that could prevent the supervisor from starting within the timeout

# Before submit, confirm executor/controller version parity:
argo version --short
kubectl -n argo get deploy workflow-controller -o jsonpath='{.spec.template.spec.containers[0].image}'
# versions must match; init-less mode is beta and protocol-sensitive

if err := waitForSupervisorReady(ctx); err != nil {
    if strings.Contains(err.Error(), "presumed dead") {
        logger.Error(ctx, "supervisor never became ready; check supervisor logs and image skew", err)
    }
    return err
}

When it happens

Trigger: evaluateSupervisorStatus sees os.Stat(statusPath) return IsNotExist continuously past the timeout — the supervisor binary never started or never wrote its first status marker.

Common situations: argoexec supervisor crashing at startup (bad image, missing template file); supervisor heartbeat goroutine dead; extremely slow node/kubelet starving the supervisor past the timeout; misbuilt image where the supervisor entrypoint is broken.

Related errors

• watching supervisor status: %w
• stat supervisor status: %w
• supervisor reported pre-main failure: %s
• supervisor presumed dead: no status update within %s
• failed to open stdout: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b3036d7ba8fbcab7. Report an issue: GitHub.