argoproj/argo-workflows · error

supervisor presumed dead: no status update within %s

Error message

supervisor presumed dead: no status update within %s

What it means

If the supervisor status marker exists but is stuck in a non-terminal state (RUNNING or a partial read) and its mtime has not advanced within the heartbeat timeout, argoexec concludes the supervisor stopped heartbeating and is presumed dead. The wait fails with this error rather than hanging until the pod deadline.

Source

Thrown at cmd/argoexec/commands/emissary.go:547

	if readErr != nil {
		// Stat just succeeded, so a read failure here means we raced the
		// supervisor's atomic rename (the old inode vanished between stat and
		// read). Treat it as transient and re-evaluate on the next tick/event
		// rather than failing the wait.
		//nolint:nilerr // deliberate: swallow the transient read error and retry
		return false, nil
	}
	token, message := parseSupervisorStatus(body)
	switch token {
	case statusReady:
		return true, nil
	case statusFailed:
		return true, fmt.Errorf("supervisor reported pre-main failure: %s", message)
	default:
		// RUNNING, or a transient/partial read: the supervisor is alive only if
		// it is still heartbeating, i.e. the marker's mtime is fresh.
		if time.Since(fi.ModTime()) > timeout {
			return true, fmt.Errorf("supervisor presumed dead: no status update within %s", timeout)
		}
		return false, nil
	}
}

// parseSupervisorStatus splits the marker into its first-line token and the
// remaining message (used by the FAILED token to carry the cause).
func parseSupervisorStatus(body []byte) (token, message string) {
	first, rest, _ := strings.Cut(string(body), "\n")
	return strings.TrimSpace(first), strings.TrimSpace(rest)
}

// waitForDependencyExitCode blocks until the given dependency exitcode file is
// written, or until a SIGTERM/SIGKILL signal is received. It uses inotify on
// the parent directory rather than polling.
//
// We deliberately do not select on ctx.Done() here: argoexec's root context is
// bound to SIGTERM via signal.NotifyContext in main.go, so when SIGTERM

View on GitHub (pinned to 35bff19146)

Solutions

1. Check supervisor logs and pod events for OOM kills or crashes around the failure time
2. Verify controller/executor version compatibility — heartbeat protocol mismatches can silence updates
3. Check node load/kubelet eviction events; reschedule onto a less loaded node
4. Re-run the workflow; if reproducible, capture the marker contents (if any) and file an issue with argoexec logs

# Confirm supervisor heartbeat health on similar nodes:
kubectl top nodes   # avoid saturated nodes
kubectl get events --field-selector type=Warning | grep -i oom  # prior OOM kills
# Executor and controller versions must match:
argo version --short

if err := waitForSupervisorReady(ctx); err != nil {
    if strings.Contains(err.Error(), "no status update within") {
        log.Printf("supervisor heartbeats stopped; check for OOM/crash and node pressure")
    }
    return err
}

When it happens

Trigger: evaluateSupervisorStatus reads a marker whose token is RUNNING (or unrecognized/partial) and time.Since(fi.ModTime()) exceeds supervisorHeartbeatTimeout — heartbeats stopped.

Common situations: Supervisor goroutine deadlocked or OOM-killed mid-run; node under heavy CPU pressure stalling heartbeats; supervisor crashed while main is still running; heartbeat interval misconfigured relative to a very short timeout.

Related errors

• watching supervisor status: %w
• supervisor presumed dead: status marker never appeared withi
• stat supervisor status: %w
• supervisor reported pre-main failure: %s
• failed to open stdout: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/a1efc7f76065f94c. Report an issue: GitHub.