argoproj/argo-workflows · error

failed to rename marker %s: %w

Error message

failed to rename marker %s: %w

What it means

writeStatusMarkerAt atomically publishes the status marker via os.Rename(tmp, path) after writing the tmp file. This error means the rename failed, so either no marker or only a stale `.tmp` file exists at the marker path. Without a valid marker the controller cannot read the step's exit status from /var/run/argo.

Source

Thrown at cmd/argoexec/commands/supervisor.go:261

	if err := writeStatusMarkerAt(path, []byte(body)); err != nil {
		logging.RequireLoggerFromContext(ctx).WithError(err).Error(ctx, "failed to write failure status marker")
	}
}

// writeStatusMarkerAt writes body to path via write-then-rename so a reader
// watching path (via inotify) only ever observes the fully-written file — it
// never sees a torn terminal message. It is the path-parameterized form used by
// tests; production calls go through writeRunningStatus / writeSuccessStatus /
// writeFailureStatus with the constant. We deliberately do not fsync — the
// marker lives in an emptyDir and the pod is gone if the node crashes before
// the write hits disk.
func writeStatusMarkerAt(path string, body []byte) error {
	tmp := path + ".tmp"
	if err := os.WriteFile(tmp, body, 0o644); err != nil {
		return fmt.Errorf("failed to write marker tmp %s: %w", tmp, err)
	}
	if err := os.Rename(tmp, path); err != nil {
		return fmt.Errorf("failed to rename marker %s: %w", path, err)
	}
	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Inspect the wrapped errno: ENOENT means the directory is gone — recreate/verify the /var/run/argo volume mount; EXDEV means cross-device rename — keep tmp and final path on the same filesystem.
2. Ensure the marker directory exists and is writable before writing (mkdir with correct perms).
3. Fix node-level issues (disk pressure, volume remount) by rescheduling the pod on a healthy node.
4. Re-run the workflow step; the marker write is transient state and succeeds once the volume is stable.

Example fix

// before: marker path on a different mount than tmp causes EXDEV
// after: keep both on the same filesystem
// tmp := filepath.Join(filepath.Dir(path), filepath.Base(path)+".tmp")

// verify dir exists and same-filesystem rename is possible
if st, err := os.Stat(filepath.Dir(path)); err != nil || !st.IsDir() {
    return fmt.Errorf("marker dir missing: %w", err)
}

func isRenameErr(err error) bool {
    var le *os.LinkError
    return errors.As(err, &le) && (errors.Is(err, syscall.ENOENT) || errors.Is(err, syscall.EXDEV))
}

if err := writeStatusMarkerAt(path, body); err != nil {
    var le *os.LinkError
    if errors.As(err, &le) && errors.Is(err, syscall.EXDEV) {
        logger.Warn(ctx, "marker rename crossed devices; check volume layout")
    }
    return err
}

When it happens

Trigger: os.Rename(path+".tmp", path) fails during writeRunningStatusAt/writeSuccessStatusAt/writeFailureStatusAt — typically because the target directory vanished, the volume was remounted read-only, or cross-device rename if path is on a different mount than tmp.

Common situations: emptyDir unmounted mid-pod (kubelet eviction/restart); node under disk pressure; misconfigured pod specs moving the marker path onto another filesystem; container runtime issues on the node.

Related errors

• failed to read container args file %s: %w
• failed to read template: %w
• failed to write marker tmp %s: %w
• failed to unmarshal container args: %w
• failed to write large arg %d to file: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/6e568c346a85679c. Report an issue: GitHub.