{"record":{"id":"82ccea7740c81836","repo":"argoproj/argo-workflows","slug":"failed-to-create-ctr-directory-w","errorCode":null,"errorMessage":"failed to create ctr directory: %w","messagePattern":"failed to create ctr directory: %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":96,"sourceCode":"\t\t\tlogger.WithError(deferErr).Error(ctx, \"Failed to shutdown tracing\")\n\t\t}\n\t}()\n\n\tctx = tracing.InjectTraceContext(ctx)\n\tworkflowName := os.Getenv(common.EnvVarWorkflowName)\n\tnamespace, _ := os.ReadFile(\"/var/run/secrets/kubernetes.io/serviceaccount/namespace\")\n\tctx, span := tracer.StartRunMainContainer(ctx, workflowName, string(namespace))\n\tdefer span.End()\n\tinjectTraceParent(ctx)\n\n\tosspecific.AllowGrantingAccessToEveryone()\n\n\t// Dir permission set to rwxrwxrwx, so that non-root wait container can also write kill signal to the folder.\n\t// Note it's important varRunArgo+\"/ctr/\" folder is writable by all, because multiple containers may want to\n\t// write to it with different users.\n\t// This also indicates we've started.\n\tif err = os.MkdirAll(varRunArgo+\"/ctr/\"+containerName, 0o777); err != nil {\n\t\treturn fmt.Errorf(\"failed to create ctr directory: %w\", err)\n\t}\n\n\tname, args := args[0], args[1:]\n\n\t// Check if args were offloaded to a file (for large args that exceed exec limit)\n\tif argsFile := os.Getenv(common.EnvVarContainerArgsFile); argsFile != \"\" {\n\t\tlogger.WithField(\"argsFile\", argsFile).Info(ctx, \"Reading container args from file\")\n\t\targsData, readErr := os.ReadFile(argsFile)\n\t\tif readErr != nil {\n\t\t\treturn fmt.Errorf(\"failed to read container args file %s: %w\", argsFile, readErr)\n\t\t}\n\t\tvar fileArgs []string\n\t\tif err = json.Unmarshal(argsData, &fileArgs); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to unmarshal container args: %w\", err)\n\t\t}\n\t\targs = append(args, fileArgs...)\n\t\tlogger.WithField(\"count\", len(fileArgs)).Info(ctx, \"Loaded container args from file\")\n","sourceCodeStart":78,"sourceCodeEnd":114,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L78-L114","documentation":"The emissary executor (argoexec emissary, PID 1 in workflow containers) creates `/var/run/argo/ctr/<containerName>` with mode 0777 so all containers (including non-root wait containers) can write exit codes and kill signals. If os.MkdirAll fails, runEmissary returns `failed to create ctr directory: %w`. This is a fatal executor startup failure — the container command never runs.","triggerScenarios":"/var/run/argo does not exist or is read-only in the container (missing emptyDir volume mount from the pod spec, or read-only rootfs); disk full; permission denied creating the dir under a restrictive security context.","commonSituations":"Custom pod specs or security policies (readOnlyRootFilesystem, restricted PSP/PSA) stripping the /var/run/argo volume; tampered workflow pod specs not created by the controller; nodes under disk pressure.","solutions":["Check the wrapped cause: EACCES vs EROFS vs ENOSPC.","Ensure the pod template includes the /var/run/argo emptyDir volume mount (workflows must not override/remove it).","If readOnlyRootFilesystem or a restricted security policy is enforced, keep the /var/run/argo mount writable (emptyDir is fine) and adjust the policy.","Check node disk space if the error is ENOSPC."],"exampleFix":"// before\nsecurityContext:\n  readOnlyRootFilesystem: true   # and no /var/run/argo volume\n// after\nvolumes:\n  - name: var-run-argo\n    emptyDir: {}\nvolumeMounts:\n  - name: var-run-argo\n    mountPath: /var/run/argo","handlingStrategy":"try-catch","validationCode":"if err := os.MkdirAll(\"/var/run/argo/ctr/test\", 0o777); err != nil { /* /var/run/argo not writable: check volume mounts & securityContext */ }","typeGuard":null,"tryCatchPattern":"if err := os.MkdirAll(varRunArgo+\"/ctr/\"+containerName, 0o777); err != nil {\n    return fmt.Errorf(\"failed to create ctr directory (is /var/run/argo mounted writable?): %w\", err)\n}","preventionTips":["Never strip the /var/run/argo emptyDir mount from workflow pod templates","Avoid readOnlyRootFilesystem without a writable /var/run/argo mount","Keep /var/run/argo writable by all users (0777) as the executor expects","Monitor nodes for disk pressure (ENOSPC)"],"tags":["argoexec","emissary","filesystem","executor-startup"],"backgroundTag":"volume-mount-failure","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}