{"record":{"id":"7b67e06983194a45","repo":"argoproj/argo-workflows","slug":"failed-to-create-dependency-dir-w","errorCode":null,"errorMessage":"failed to create dependency dir: %w","messagePattern":"failed to create dependency dir: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":194,"sourceCode":"\t\t\treturn argoerrors.NewExitErrWithCause(exitCode, stageErr)\n\t\t}\n\t}\n\n\t// setup signal handlers\n\tsignals := make(chan os.Signal, 1)\n\tdefer close(signals)\n\tsignal.Notify(signals)\n\tdefer signal.Reset()\n\n\tfor _, x := range template.ContainerSet.GetGraph() {\n\t\tif x.Name == containerName {\n\t\t\tfor _, y := range x.Dependencies {\n\t\t\t\tlogger.WithField(\"dependency\", y).Info(ctx, \"waiting for dependency\")\n\t\t\t\tdepDir := filepath.Clean(varRunArgo + \"/ctr/\" + y)\n\t\t\t\t// The dependency container will MkdirAll this too, but may not have\n\t\t\t\t// started yet; pre-create it so we can install an inotify watch on it.\n\t\t\t\tif err = os.MkdirAll(depDir, 0o777); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to create dependency dir: %w\", err)\n\t\t\t\t}\n\t\t\t\tdepExitPath := filepath.Join(depDir, \"exitcode\")\n\t\t\t\tcode, waitErr := waitForDependencyExitCode(ctx, depExitPath, signals)\n\t\t\t\tif waitErr != nil {\n\t\t\t\t\treturn waitErr\n\t\t\t\t}\n\t\t\t\texitCode = code\n\t\t\t\tif exitCode != 0 {\n\t\t\t\t\treturn fmt.Errorf(\"dependency %q exited with non-zero code: %d\", y, exitCode)\n\t\t\t\t}\n\t\t\t}\n\t\t}\n\t}\n\n\tname, err = exec.LookPath(name)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"failed to find name in PATH: %w\", err)\n\t}","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L176-L212","documentation":"argoexec's emissary PID-1 wraps each container in a workflow pod. When the container template declares dependencies on other containers (container sets), runEmissary pre-creates /var/run/argo/ctr/<dep> for each dependency so an inotify watch can be installed before the dependency container starts. If os.MkdirAll fails (mkdir of the path failed for a reason other than the dir already existing), this error wraps that OS error.","triggerScenarios":"os.MkdirAll on /var/run/argo/ctr/<dependencyName> fails: the parent /var/run/argo volume is not mounted or is read-only, a non-directory file already exists at the path, or the filesystem returned EPERM/EACCES/ENOSPC.","commonSituations":"Running argoexec in a pod where the argo run-artifacts emptyDir/volume for /var/run/argo is missing or mounted read-only; a file named like the dependency container exists at /var/run/argo/ctr/; disk-full nodes; securityContext restricting writes to /var/run/argo.","solutions":["Verify the /var/run/argo volume is mounted writable in the pod spec (workflow controller injects it; do not mark it readOnly)","Check nothing but directories exists under /var/run/argo/ctr — delete any stale file named like the dependency container","Check node disk space and pod securityContext (fsGroup/runAsUser) allows writes to /var/run/argo","Inspect the wrapped OS error (%w) to distinguish EACCES vs ENOSPC vs ENOTDIR"],"exampleFix":"// before (read-only mount in a modified pod spec)\nvolumeMounts:\n- name: var-run-argo\n  mountPath: /var/run/argo\n  readOnly: true\n// after\nvolumeMounts:\n- name: var-run-argo\n  mountPath: /var/run/argo","handlingStrategy":"validation","validationCode":"const depDir = \"/var/run/argo/ctr/\" + dep\ntry { require(\"fs\").accessSync(\"/var/run/argo\", require(\"fs\").constants.W_OK) } catch { throw new Error(\"/var/run/argo not writable — check volume mounts\") }\nif (require(\"fs\").existsSync(depDir) && !require(\"fs\").statSync(depDir).isDirectory()) throw new Error(depDir + \" exists and is not a directory\")","typeGuard":"function isWritableDir(p) { try { return require('fs').statSync(p).isDirectory() && !! (require('fs').accessSync(p, require('fs').constants.W_OK), true) } catch { return false } }","tryCatchPattern":"try { await runEmissary(...) } catch (e) { if (/failed to create dependency dir/.test(e.message)) { checkVolumeMounts(); } throw e }","preventionTips":["Never mark the /var/run/argo volume readOnly in container-set workflows","Keep container names filesystem-safe (no slashes/special chars)","Monitor node disk usage; alert on ENOSPC in executor logs","Test workflows that use container dependencies on your actual securityContext"],"tags":["kubernetes","argoexec","filesystem","container-set"],"backgroundTag":"mkdir-permission-denied","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}