argoproj/argo-workflows · error

failed to stat input artifact %q at %s: %w

Error message

failed to stat input artifact %q at %s: %w

What it means

For each input artifact, linkInputArtifactsAt Lstats the supervisor-written source at <baseDir>/<artifact-name> before symlinking it to the artifact's declared path. Missing sources are treated as optional and skipped, but any other stat error (EACCES, ELOOP, EIO) is fatal and wrapped as this error. It means the executor could not even inspect the staged artifact entry.

Source

Thrown at cmd/argoexec/commands/emissary.go:391

// content. Code that calls `lstat`/`readlink` on art.Path will observe a
// symlink rather than a regular file. `rm art.Path` removes the symlink
// only; the underlying artifact stays in the shared emptyDir.
//
// Overlapping user volumes are handled by the executor on the write side
// (supervisor writes to /mainctrfs/<art.Path> instead of /argo/inputs/
// artifacts/<name>), so no entry appears in the input-artifacts directory
// and we skip the symlink — main already sees the file at art.Path via
// its user volume.
func linkInputArtifactsAt(ctx context.Context, baseDir string, tmpl *wfv1.Template) error {
	logger := logging.RequireLoggerFromContext(ctx)
	for _, art := range tmpl.Inputs.Artifacts {
		src := filepath.Join(baseDir, art.Name)
		if _, statErr := os.Lstat(src); statErr != nil {
			if os.IsNotExist(statErr) {
				logger.WithFields(logging.Fields{"name": art.Name, "path": art.Path}).Info(ctx, "no input-artifacts entry (optional or overlap) — skipping symlink")
				continue
			}
			return fmt.Errorf("failed to stat input artifact %q at %s: %w", art.Name, src, statErr)
		}
		dst := art.Path
		if dst == "" {
			continue
		}
		if parent := filepath.Dir(dst); parent != "" && parent != "/" {
			if err := os.MkdirAll(parent, 0o755); err != nil {
				return fmt.Errorf("failed to create parent directory for artifact %q at %s: %w", art.Name, dst, err)
			}
		}
		// If nothing exists at art.Path, just create the symlink. Creating is
		// always safe — os.Symlink returns EEXIST rather than overwriting and the
		// MkdirAll above only ever creates — so even when art.Path resolves into a
		// user volume we deliberately let the artifact land there (the user asked
		// for it). Only an *overwrite* can destroy data, and that is gated below.
		if _, err := os.Lstat(dst); err != nil {
			if !os.IsNotExist(err) {
				return fmt.Errorf("failed to stat artifact path %q at %s: %w", art.Name, dst, err)

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped inner error to distinguish EACCES (permissions) from EIO (disk).
2. Fix the executor container's securityContext so /var/run/argo is readable/writable.
3. Check the node's disk health and events (kubectl describe node) for I/O pressure.
4. If the artifact genuinely wasn't staged, verify the init/supervisor artifact-staging logs — a missing entry would normally be skipped, so a persistent stat error implies filesystem trouble.
5. Resubmit the workflow on a different node to rule out node-local disk corruption.

// Verify the staged artifact source is reachable before linking:
test -e /argo/inputs/artifacts/<name> || echo "not staged (optional?)"

try {
  await stageArtifacts(tmpl)
} catch (e) {
  if (String(e).includes('failed to stat input artifact')) {
    if (isTransient(e)) await retryWithBackoff(stageArtifacts) // e.g. NFS EIO
    else throw e // permissions/filesystem corruption need a fix
  }
}

When it happens

Trigger: os.Lstat(src) fails with a non-ENOsport error: /argo/inputs/artifacts (or the configured baseDir) is not readable, the filesystem returned I/O error, or too many symlink levels (ELOOP) from a corrupted emptyDir.

Common situations: Pod security contexts denying read on /argo; corrupted node disk/emptyDir; node under disk pressure causing I/O errors; a user-supplied baseDir override pointing somewhere inaccessible in tests.

Related errors

• failed to create parent directory for artifact %q at %s: %w
• failed to stat artifact path %q at %s: %w
• failed to resolve parent of artifact path %q at %s: %w
• failed to clear existing path for artifact %q at %s: %w
• failed to read container args file %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/7fa5fdcf264fe0c8. Report an issue: GitHub.