argoproj/argo-workflows · error

watching supervisor status: %w

Error message

watching supervisor status: %w

What it means

In init-less pod mode, the emissary watches the supervisor's status marker with an inotify-based file watcher. If the watcher itself fails (its WatchFile call returns an error) while the watch context is still active, the emissary finishes the wait with this wrapped error rather than hanging.

Source

Thrown at cmd/argoexec/commands/emissary.go:490

	resCh := make(chan error, 1)
	finish := func(err error) {
		select {
		case resCh <- err:
		default: // a result already landed; first one wins
		}
		cancel()
	}

	// Low-latency terminal detection: re-evaluate on every write to the marker
	// (heartbeats and the terminal write both fire here).
	go func() {
		werr := file.WatchFile(watchCtx, statusPath, func() {
			if done, err := evaluateSupervisorStatus(statusPath, timeout, start); done {
				finish(err)
			}
		})
		if werr != nil && watchCtx.Err() == nil {
			finish(fmt.Errorf("watching supervisor status: %w", werr))
		}
	}()

	ticker := time.NewTicker(pollInterval)
	defer ticker.Stop()
	for {
		select {
		case <-ctx.Done():
			return ctx.Err()
		case err := <-resCh:
			if err == nil {
				logger.Info(ctx, "supervisor is ready")
			}
			return err
		case <-ticker.C:
			if done, err := evaluateSupervisorStatus(statusPath, timeout, start); done {
				finish(err)
			}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped underlying error for the exact watcher failure
2. Raise the host inotify limits: sysctl fs.inotify.max_user_watches/max_user_instances (node-level, needs admin)
3. Ensure nothing in the pod deletes /var/run/argo/ctr/<name> while the workflow runs
4. Re-run the workflow; if transient it may succeed — if persistent, capture argoexec logs and node sysctl values

# On the node hosting workflows, verify inotify capacity:
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances
# Raise if exhausted (node-level):
sudo sysctl -w fs.inotify.max_user_watches=524288

if err := waitForSupervisorReady(ctx); err != nil {
    if strings.Contains(err.Error(), "watching supervisor status") {
        // transient watcher failure: retry once before failing the pod
        time.Sleep(pollInterval)
        return waitForSupervisorReady(ctx)
    }
    return err
}

When it happens

Trigger: file.WatchFile returns an error while waiting for the supervisor status marker — e.g. the watched directory or file was removed, inotify limits were exhausted, or the watch could not be established on /var/run/argo/ctr/<name>.

Common situations: Host inotify watch limit exhausted (fs.inotify.max_user_watches); /var/run/argo directory cleaned up by another process; kernel/container environment lacking inotify support.

Related errors

• supervisor presumed dead: status marker never appeared withi
• stat supervisor status: %w
• supervisor reported pre-main failure: %s
• supervisor presumed dead: no status update within %s
• failed to open stdout: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/da3fda9237ccaa96. Report an issue: GitHub.