argoproj/argo-workflows · critical

failed to create dependency dir: %w

Error message

failed to create dependency dir: %w

What it means

argoexec's emissary PID-1 wraps each container in a workflow pod. When the container template declares dependencies on other containers (container sets), runEmissary pre-creates /var/run/argo/ctr/<dep> for each dependency so an inotify watch can be installed before the dependency container starts. If os.MkdirAll fails (mkdir of the path failed for a reason other than the dir already existing), this error wraps that OS error.

Source

Thrown at cmd/argoexec/commands/emissary.go:194

			return argoerrors.NewExitErrWithCause(exitCode, stageErr)
		}
	}

	// setup signal handlers
	signals := make(chan os.Signal, 1)
	defer close(signals)
	signal.Notify(signals)
	defer signal.Reset()

	for _, x := range template.ContainerSet.GetGraph() {
		if x.Name == containerName {
			for _, y := range x.Dependencies {
				logger.WithField("dependency", y).Info(ctx, "waiting for dependency")
				depDir := filepath.Clean(varRunArgo + "/ctr/" + y)
				// The dependency container will MkdirAll this too, but may not have
				// started yet; pre-create it so we can install an inotify watch on it.
				if err = os.MkdirAll(depDir, 0o777); err != nil {
					return fmt.Errorf("failed to create dependency dir: %w", err)
				}
				depExitPath := filepath.Join(depDir, "exitcode")
				code, waitErr := waitForDependencyExitCode(ctx, depExitPath, signals)
				if waitErr != nil {
					return waitErr
				}
				exitCode = code
				if exitCode != 0 {
					return fmt.Errorf("dependency %q exited with non-zero code: %d", y, exitCode)
				}
			}
		}
	}

	name, err = exec.LookPath(name)
	if err != nil {
		return fmt.Errorf("failed to find name in PATH: %w", err)
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Verify the /var/run/argo volume is mounted writable in the pod spec (workflow controller injects it; do not mark it readOnly)
2. Check nothing but directories exists under /var/run/argo/ctr — delete any stale file named like the dependency container
3. Check node disk space and pod securityContext (fsGroup/runAsUser) allows writes to /var/run/argo
4. Inspect the wrapped OS error (%w) to distinguish EACCES vs ENOSPC vs ENOTDIR

Example fix

// before (read-only mount in a modified pod spec)
volumeMounts:
- name: var-run-argo
  mountPath: /var/run/argo
  readOnly: true
// after
volumeMounts:
- name: var-run-argo
  mountPath: /var/run/argo

const depDir = "/var/run/argo/ctr/" + dep
try { require("fs").accessSync("/var/run/argo", require("fs").constants.W_OK) } catch { throw new Error("/var/run/argo not writable — check volume mounts") }
if (require("fs").existsSync(depDir) && !require("fs").statSync(depDir).isDirectory()) throw new Error(depDir + " exists and is not a directory")

function isWritableDir(p) { try { return require('fs').statSync(p).isDirectory() && !! (require('fs').accessSync(p, require('fs').constants.W_OK), true) } catch { return false } }

try { await runEmissary(...) } catch (e) { if (/failed to create dependency dir/.test(e.message)) { checkVolumeMounts(); } throw e }

When it happens

Trigger: os.MkdirAll on /var/run/argo/ctr/<dependencyName> fails: the parent /var/run/argo volume is not mounted or is read-only, a non-directory file already exists at the path, or the filesystem returned EPERM/EACCES/ENOSPC.

Common situations: Running argoexec in a pod where the argo run-artifacts emptyDir/volume for /var/run/argo is missing or mounted read-only; a file named like the dependency container exists at /var/run/argo/ctr/; disk-full nodes; securityContext restricting writes to /var/run/argo.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• dependency %q exited with non-zero code: %d
• stat supervisor status: %w
• failed to open stdout: %w
• failed to open combined: %w
• failed to create ctr directory: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/7b67e06983194a45. Report an issue: GitHub.