argoproj/argo-workflows · error

failed to leave working directory before staging input artif

Error message

failed to leave working directory before staging input artifacts: %w

What it means

To stage input artifacts safely, argoexec temporarily chdirs to /var/run/argo (so symlinks can be created without a deleted-cwd problem) and later chdirs back to the original directory. This error means the temporary chdir into /var/run/argo failed, so artifact staging is aborted before anything was linked.

Source

Thrown at cmd/argoexec/commands/emissary.go:353

func stageInputArtifacts(ctx context.Context, tmpl *wfv1.Template) error {
	return stageInputArtifactsAt(ctx, common.ExecutorArtifactBaseDir, tmpl)
}

// stageInputArtifactsAt is the parameterized form used by tests; production
// calls stageInputArtifacts with the constants. It links each input artifact
// into place and re-enters the working directory afterwards, stepping off it
// for the duration: linking replaces the cwd when an artifact's path is the
// container's workingDir, Windows refuses to delete a directory in use as a
// working directory, and a child forked with the deleted directory as cwd
// would see getcwd() fail and relative paths resolve to nothing. The final
// chdir follows the symlink to whatever now sits at the path.
func stageInputArtifactsAt(ctx context.Context, baseDir string, tmpl *wfv1.Template) error {
	origWd, err := os.Getwd()
	if err != nil {
		return fmt.Errorf("failed to read working directory before staging input artifacts: %w", err)
	}
	if err := os.Chdir(varRunArgo); err != nil {
		return fmt.Errorf("failed to leave working directory before staging input artifacts: %w", err)
	}
	if err := linkInputArtifactsAt(ctx, baseDir, tmpl); err != nil {
		return err
	}
	if err := os.Chdir(origWd); err != nil {
		return fmt.Errorf("failed to re-enter working directory %q after staging input artifacts (an input artifact staged at the workingDir path must be a directory): %w", origWd, err)
	}
	return nil
}

// linkInputArtifactsAt creates a symlink at each input artifact's path
// pointing to the file that supervisor wrote under /argo/inputs/artifacts/
// <name>. This replaces the legacy SubPath bind-mount-per-artifact scheme,
// which can't be used in init-less mode because kubelet pre-creates SubPath
// entries as empty directories before supervisor can write the real file.
//
// Behavior notes for workflow authors: in init-less mode art.Path is a
// symlink rather than a regular file. `cat`, `open()`, `tar`, `cp`,

View on GitHub (pinned to 35bff19146)

Solutions

1. Verify /var/run/argo exists and is writable inside the executor container.
2. Check the pod's securityContext and any PSA/OPA policies restricting filesystem writes.
3. Ensure the pod spec includes the standard /var/run/argo emptyDir volume.
4. Read the wrapped errno (ENOENT vs EACCES vs EROFS) to pinpoint the cause.
5. Use the stock executor pod spec from the installed Argo version instead of a hand-modified one.

Example fix

// before (read-only mount in custom podspec)
- name: argo-run
  emptyDir: {}
  # mounted readOnly: true
// after
- name: argo-run
  emptyDir: {}   # mounted normally (writable)

// Verify /var/run/argo is writable before staging:
test -d /var/run/argo && test -w /var/run/argo || { echo "/var/run/argo not writable" >&2; exit 1; }

try {
  await stageArtifacts(tmpl)
} catch (e) {
  if (String(e).includes('failed to leave working directory')) {
    // check securityContext / volume mount readOnly flags
  }
}

When it happens

Trigger: os.Chdir(varRunArgo) returns an error — the /var/run/argo directory does not exist (volume mount removed/renamed) or the process lacks execute permission on it (restricted pod security standards, read-only tmpfs).

Common situations: Custom pod specs that mount /var/run/argo read-only; hardened securityContext (no root, restricted filesystem) blocking access; controller/executor version mismatch where the emptyDir was never created; running argoexec locally where /var/run/argo doesn't exist.

Understand the failure class

Background: Permission denied / not authorized / 403 Forbidden: access-control rejections when the caller lacks the required role, grant, or ownership — this error's family across 18 libraries.

Related errors

• failed to create parent directory for artifact %q at %s: %w
• failed to clear existing path for artifact %q at %s: %w
• failed to read container args file %s: %w
• failed to write large arg %d to file: %w
• failed to read template: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b6544de878d8b4ba. Report an issue: GitHub.