argoproj/argo-workflows · error

failed to write marker tmp %s: %w

Error message

failed to write marker tmp %s: %w

What it means

writeStatusMarkerAt writes the argoexec status marker (running/success/failure) to a path under /var/run/argo by first writing a `.tmp` sidecar file and atomically renaming it. This error wraps a failure of os.WriteFile on the tmp file, so the status marker was never produced. The comment notes fsync is deliberately skipped since the marker lives in an emptyDir that dies with the pod.

Source

Thrown at cmd/argoexec/commands/supervisor.go:258

	if msg := cause.Error(); msg != "" {
		body += "\n" + msg
	}
	if err := writeStatusMarkerAt(path, []byte(body)); err != nil {
		logging.RequireLoggerFromContext(ctx).WithError(err).Error(ctx, "failed to write failure status marker")
	}
}

// writeStatusMarkerAt writes body to path via write-then-rename so a reader
// watching path (via inotify) only ever observes the fully-written file — it
// never sees a torn terminal message. It is the path-parameterized form used by
// tests; production calls go through writeRunningStatus / writeSuccessStatus /
// writeFailureStatus with the constant. We deliberately do not fsync — the
// marker lives in an emptyDir and the pod is gone if the node crashes before
// the write hits disk.
func writeStatusMarkerAt(path string, body []byte) error {
	tmp := path + ".tmp"
	if err := os.WriteFile(tmp, body, 0o644); err != nil {
		return fmt.Errorf("failed to write marker tmp %s: %w", tmp, err)
	}
	if err := os.Rename(tmp, path); err != nil {
		return fmt.Errorf("failed to rename marker %s: %w", path, err)
	}
	return nil
}

View on GitHub (pinned to 35bff19146)

Solutions

1. Check the wrapped error (`%w`) for ENOSPC/EACCES/EROFS and fix the underlying volume issue (free space, correct mount).
2. Verify the pod spec still mounts an emptyDir (or equivalent writable volume) at /var/run/argo and the argoexec container can write to it.
3. Confirm the container securityContext does not set readOnlyRootFilesystem without a writable volume for /var/run/argo.
4. Re-run the workflow pod on a healthy node to rule out node-level disk or permission problems.

Example fix

// before: container silently missing the writable mount
// after: ensure the mount exists
// containers:
//   - name: main
//     volumeMounts:
//       - name: var-run-argo
//         mountPath: /var/run/argo
// volumes:
//   - name: var-run-argo
//     emptyDir: {}

// preflight in Go
if st, err := os.Stat(filepath.Dir(path)); err != nil || !st.IsDir() {
    return fmt.Errorf("marker dir %s not usable: %w", path, err)
}
if f, err := os.OpenFile(path+".tmp", os.O_CREATE|os.O_WRONLY, 0o644); err != nil {
    return fmt.Errorf("marker path not writable: %w", err)
} else { f.Close(); os.Remove(path + ".tmp") }

func isMarkerWriteErr(err error) bool {
    var pe *fs.PathError
    return errors.As(err, &pe) && (errors.Is(err, fs.ErrPermission) || errors.Is(err, syscall.ENOSPC) || errors.Is(err, syscall.EROFS))
}

if err := writeStatusMarkerAt(path, body); err != nil {
    var pe *fs.PathError
    if errors.As(err, &pe) && errors.Is(err, syscall.ENOSPC) {
        logger.Warn(ctx, "disk full, marker not written")
    } else {
        logger.Error(ctx, "status marker write failed: %v", err)
    }
    return err
}

When it happens

Trigger: Called by writeRunningStatusAt, writeSuccessStatusAt, or writeFailureStatusAt when os.WriteFile(path+".tmp", body, 0o644) fails — e.g. the emptyDir volume backing the marker path is full, read-only, or the /var/run/argo directory does not exist or has wrong permissions.

Common situations: Disk-pressure evicted or volume-full nodes; security contexts that make the container filesystem read-only; modified volume mounts that drop the /var/run/argo emptyDir; SELinux/AppArmor blocking writes to the mount.

Related errors

• failed to read container args file %s: %w
• failed to read template: %w
• failed to rename marker %s: %w
• failed to unmarshal container args: %w
• failed to write large arg %d to file: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/d4d4640351279735. Report an issue: GitHub.