{"record":{"id":"6e568c346a85679c","repo":"argoproj/argo-workflows","slug":"failed-to-rename-marker-s-w","errorCode":null,"errorMessage":"failed to rename marker %s: %w","messagePattern":"failed to rename marker (.+?): %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/argoexec/commands/supervisor.go","lineNumber":261,"sourceCode":"\tif err := writeStatusMarkerAt(path, []byte(body)); err != nil {\n\t\tlogging.RequireLoggerFromContext(ctx).WithError(err).Error(ctx, \"failed to write failure status marker\")\n\t}\n}\n\n// writeStatusMarkerAt writes body to path via write-then-rename so a reader\n// watching path (via inotify) only ever observes the fully-written file — it\n// never sees a torn terminal message. It is the path-parameterized form used by\n// tests; production calls go through writeRunningStatus / writeSuccessStatus /\n// writeFailureStatus with the constant. We deliberately do not fsync — the\n// marker lives in an emptyDir and the pod is gone if the node crashes before\n// the write hits disk.\nfunc writeStatusMarkerAt(path string, body []byte) error {\n\ttmp := path + \".tmp\"\n\tif err := os.WriteFile(tmp, body, 0o644); err != nil {\n\t\treturn fmt.Errorf(\"failed to write marker tmp %s: %w\", tmp, err)\n\t}\n\tif err := os.Rename(tmp, path); err != nil {\n\t\treturn fmt.Errorf(\"failed to rename marker %s: %w\", path, err)\n\t}\n\treturn nil\n}\n","sourceCodeStart":243,"sourceCodeEnd":265,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/supervisor.go#L243-L265","documentation":"writeStatusMarkerAt atomically publishes the status marker via os.Rename(tmp, path) after writing the tmp file. This error means the rename failed, so either no marker or only a stale `.tmp` file exists at the marker path. Without a valid marker the controller cannot read the step's exit status from /var/run/argo.","triggerScenarios":"os.Rename(path+\".tmp\", path) fails during writeRunningStatusAt/writeSuccessStatusAt/writeFailureStatusAt — typically because the target directory vanished, the volume was remounted read-only, or cross-device rename if path is on a different mount than tmp.","commonSituations":"emptyDir unmounted mid-pod (kubelet eviction/restart); node under disk pressure; misconfigured pod specs moving the marker path onto another filesystem; container runtime issues on the node.","solutions":["Inspect the wrapped errno: ENOENT means the directory is gone — recreate/verify the /var/run/argo volume mount; EXDEV means cross-device rename — keep tmp and final path on the same filesystem.","Ensure the marker directory exists and is writable before writing (mkdir with correct perms).","Fix node-level issues (disk pressure, volume remount) by rescheduling the pod on a healthy node.","Re-run the workflow step; the marker write is transient state and succeeds once the volume is stable."],"exampleFix":"// before: marker path on a different mount than tmp causes EXDEV\n// after: keep both on the same filesystem\n// tmp := filepath.Join(filepath.Dir(path), filepath.Base(path)+\".tmp\")","handlingStrategy":"try-catch","validationCode":"// verify dir exists and same-filesystem rename is possible\nif st, err := os.Stat(filepath.Dir(path)); err != nil || !st.IsDir() {\n    return fmt.Errorf(\"marker dir missing: %w\", err)\n}","typeGuard":"func isRenameErr(err error) bool {\n    var le *os.LinkError\n    return errors.As(err, &le) && (errors.Is(err, syscall.ENOENT) || errors.Is(err, syscall.EXDEV))\n}","tryCatchPattern":"if err := writeStatusMarkerAt(path, body); err != nil {\n    var le *os.LinkError\n    if errors.As(err, &le) && errors.Is(err, syscall.EXDEV) {\n        logger.Warn(ctx, \"marker rename crossed devices; check volume layout\")\n    }\n    return err\n}","preventionTips":["Ensure the marker directory persists for the pod lifetime (emptyDir, not a deleted path).","Never place tmp and target on different mounts (EXDEV).","Handle kubelet volume remounts; avoid writing markers during termination races.","Test marker writes on node-disk-pressure scenarios."],"tags":["executor","filesystem","kubernetes"],"backgroundTag":"atomic-rename-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}