argoproj/argo-workflows · error

stat supervisor status: %w

Error message

stat supervisor status: %w

What it means

When polling the supervisor status marker, a stat error other than NotExist (e.g. EACCES, ELOOP, or the containing directory vanished) is treated as fatal and returned wrapped as 'stat supervisor status'. Unlike a read failure right after stat (which is a benign rename race), a failing stat cannot be retried safely, so the wait terminates with this error.

Source

Thrown at cmd/argoexec/commands/emissary.go:526

			}
		}
	}
}

// evaluateSupervisorStatus reads the status marker once and decides whether main
// can stop waiting. done=false means keep waiting. start is main's wait-start
// reference, used to bound the case where the marker never appears at all. It is
// safe to call concurrently — it only reads the filesystem.
func evaluateSupervisorStatus(statusPath string, timeout time.Duration, start time.Time) (done bool, err error) {
	fi, statErr := os.Stat(statusPath)
	if statErr != nil {
		if os.IsNotExist(statErr) {
			if time.Since(start) > timeout {
				return true, fmt.Errorf("supervisor presumed dead: status marker never appeared within %s", timeout)
			}
			return false, nil
		}
		return true, fmt.Errorf("stat supervisor status: %w", statErr)
	}
	body, readErr := os.ReadFile(statusPath)
	if readErr != nil {
		// Stat just succeeded, so a read failure here means we raced the
		// supervisor's atomic rename (the old inode vanished between stat and
		// read). Treat it as transient and re-evaluate on the next tick/event
		// rather than failing the wait.
		//nolint:nilerr // deliberate: swallow the transient read error and retry
		return false, nil
	}
	token, message := parseSupervisorStatus(body)
	switch token {
	case statusReady:
		return true, nil
	case statusFailed:
		return true, fmt.Errorf("supervisor reported pre-main failure: %s", message)
	default:
		// RUNNING, or a transient/partial read: the supervisor is alive only if

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped errno in the full error to identify the exact stat failure
2. Ensure no sidecar/init process deletes or chmods /var/run/argo/ctr/<name> while the workflow runs
3. Run the container with a securityContext that keeps access to /var/run/argo (default should suffice; check fsGroup/readOnly tweaks)
4. Re-run the workflow; persistent occurrence warrants collecting argoexec debug logs and an issue report

# Ensure nothing mutates /var/run/argo mid-run; verify from a debug pod:
kubectl exec -it <pod> -c init -- ls -la /var/run/argo/ctr/
# The ctr/<name> directory must persist for the lifetime of the pod.

if err := waitForSupervisorReady(ctx); err != nil {
    var perr *fs.PathError
    if errors.As(err, &perr) && perr.Op == "stat" {
        log.Printf("status marker stat failed path=%s err=%v", perr.Path, perr.Err)
        // EACCES -> securityContext; ENOENT after dir removal -> rogue process
    }
    return err
}

When it happens

Trigger: os.Stat on /var/run/argo/ctr/<name>/status returns a non-IsNotExist error during waitForSupervisorReady — permission denied on the directory, path became a symlink loop, or the ctr directory was deleted mid-run.

Common situations: Something in the pod removing /var/run/argo/ctr/<container>; restrictive securityContext blocking access to /var/run/argo; corrupted tmpfs/emptyDir backing /var/run/argo.

Related errors

• watching supervisor status: %w
• supervisor presumed dead: status marker never appeared withi
• supervisor reported pre-main failure: %s
• supervisor presumed dead: no status update within %s
• failed to open stdout: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/85a9bb9594e51569. Report an issue: GitHub.