argoproj/argo-workflows · error

failed to write initial status marker: %w

Error message

failed to write initial status marker: %w

What it means

startStatusHeartbeat writes a synchronous initial "running" status marker to a path shared between the supervisor and main containers before starting a periodic heartbeat goroutine. If that initial write fails, the error is wrapped as "failed to write initial status marker" and returned so callers fail fast — a broken shared mount would otherwise leave the main container waiting on a status that never appears.

Source

Thrown at cmd/argoexec/commands/supervisor.go:190

	names := make([]wfv1.ArtifactPluginName, 0, len(raw))
	for _, p := range raw {
		names = append(names, wfv1.ArtifactPluginName(p))
	}
	return names
}

// startStatusHeartbeat writes an initial RUNNING status, then rewrites it every
// supervisorHeartbeatInterval on a background goroutine until the returned stop
// function is called. Each rewrite advances the marker's mtime, which main's
// emissary uses to distinguish a live (but slow) supervisor from a dead one.
//
// The initial write is synchronous so a broken shared mount fails fast (its
// error is returned). stop() cancels the goroutine and blocks until it has
// exited, guaranteeing no heartbeat write can race the terminal status write
// that follows it.
func startStatusHeartbeat(ctx context.Context) (stop func(), err error) {
	if err := writeRunningStatus(); err != nil {
		return nil, fmt.Errorf("failed to write initial status marker: %w", err)
	}
	hbCtx, cancel := context.WithCancel(ctx)
	done := make(chan struct{})
	go func() {
		defer close(done)
		ticker := time.NewTicker(supervisorHeartbeatInterval)
		defer ticker.Stop()
		for {
			select {
			case <-hbCtx.Done():
				return
			case <-ticker.C:
				if err := writeRunningStatus(); err != nil {
					logging.RequireLoggerFromContext(ctx).WithError(err).Warn(ctx, "failed to write status heartbeat")
				}
			}
		}
	}()

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped cause (EACCES/ENOENT/ENOSPC) to identify the filesystem problem
2. Ensure the shared status volume is mounted at the expected /var/run/argo path in the supervisor container
3. Make the mount writable by the argoexec user (fsGroup/securityContext adjustments)
4. Free storage or raise ephemeral-storage limits if ENOSPC

Example fix

// before: supervisor container missing the shared mount
containers:
  - name: main
    volumeMounts: []
// after
containers:
  - name: main
    volumeMounts:
      - name: var-run-argo
        mountPath: /var/run/argo
      # supervisor sidecar must mount the same volume

// fail fast in CI/dev with a supervisor smoke test:
// kubectl exec <pod> -c main -- test -w /var/run/argo/status-path-dir

stop, err := startStatusHeartbeat(ctx)
if err != nil {
    var pe *fs.PathError
    if errors.As(err, &pe) {
        // mount/permission problem on the shared status volume — fix before main starts
    }
    return err
}

When it happens

Trigger: A supervisor container calling startStatusHeartbeat where writeRunningStatus() returns an error on the first write — /var/run/argo shared volume missing, read-only, or full; permission denied on the status file path.

Common situations: Init-less (supervisor) workflows where the emptyDir shared between main and supervisor containers is misconfigured or not mounted in the supervisor; restricted pod security policies denying writes; disk-full nodes.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• failed to write template: %w
• failed to stage files: %w
• %w
• failed to load non-plugin input artifacts: %w
• failed to load input artifacts from plugin %q: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/845647f02541df79. Report an issue: GitHub.