{"record":{"id":"0a59ea2973fce3ff","repo":"argoproj/argo-workflows","slug":"failed-to-read-container-args-file-s-w","errorCode":null,"errorMessage":"failed to read container args file %s: %w","messagePattern":"failed to read container args file (.+?): %w","errorType":"console","errorClass":null,"httpStatus":null,"severity":"critical","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":106,"sourceCode":"\n\tosspecific.AllowGrantingAccessToEveryone()\n\n\t// Dir permission set to rwxrwxrwx, so that non-root wait container can also write kill signal to the folder.\n\t// Note it's important varRunArgo+\"/ctr/\" folder is writable by all, because multiple containers may want to\n\t// write to it with different users.\n\t// This also indicates we've started.\n\tif err = os.MkdirAll(varRunArgo+\"/ctr/\"+containerName, 0o777); err != nil {\n\t\treturn fmt.Errorf(\"failed to create ctr directory: %w\", err)\n\t}\n\n\tname, args := args[0], args[1:]\n\n\t// Check if args were offloaded to a file (for large args that exceed exec limit)\n\tif argsFile := os.Getenv(common.EnvVarContainerArgsFile); argsFile != \"\" {\n\t\tlogger.WithField(\"argsFile\", argsFile).Info(ctx, \"Reading container args from file\")\n\t\targsData, readErr := os.ReadFile(argsFile)\n\t\tif readErr != nil {\n\t\t\treturn fmt.Errorf(\"failed to read container args file %s: %w\", argsFile, readErr)\n\t\t}\n\t\tvar fileArgs []string\n\t\tif err = json.Unmarshal(argsData, &fileArgs); err != nil {\n\t\t\treturn fmt.Errorf(\"failed to unmarshal container args: %w\", err)\n\t\t}\n\t\targs = append(args, fileArgs...)\n\t\tlogger.WithField(\"count\", len(fileArgs)).Info(ctx, \"Loaded container args from file\")\n\n\t\t// Check for a large args and offload to file if needed\n\t\t// This avoids the exec() \"argument list too long\" error\n\t\t// Downstream programs should support @filename for parsing large args\n\t\tfor i := 0; i < len(args); i++ {\n\t\t\tif len(args[i]) > common.MaxEnvVarLen {\n\t\t\t\tfilePath := fmt.Sprintf(\"/tmp/argo_arg_%d.txt\", i)\n\t\t\t\tif err = os.WriteFile(filePath, []byte(args[i]), 0o644); err != nil {\n\t\t\t\t\treturn fmt.Errorf(\"failed to write large arg %d to file: %w\", i, err)\n\t\t\t\t}\n\t\t\t\tlogger.WithFields(logging.Fields{","sourceCodeStart":88,"sourceCodeEnd":124,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L88-L124","documentation":"In `argoexec emissary`, when the controller offloads oversized container args to a file (env var ARGO_CONTAINER_ARGS_FILE is set, because the marshaled args exceeded MaxEnvVarLen), runEmissary reads that file with os.ReadFile. This error is thrown when that read fails (missing file, wrong path, permission denied). Without the args the emissary cannot exec the user command, so the container fails immediately. The underlying OS error is wrapped with %w so errors.Is/As still work.","triggerScenarios":"ARGO_CONTAINER_ARGS_FILE points to a path that does not exist or is unreadable at the time emissary starts: the ConfigMap holding the offloaded args (named after the pod, created in workflowpod.go build()) was not mounted, was deleted before the container started, or the mount path in the env var does not match the volumeMount.","commonSituations":"Running with a controller version that offloads args but an executor image too old/new to agree on the env var name or mount location; a ConfigMap-name collision or garbage-collection race in a busy namespace; a custom pod spec patch that drops or renames the args ConfigMap volume; read-only or restrictive securityContext blocking the mount.","solutions":["Check the pod spec (`kubectl get pod <pod> -o yaml`): confirm the env var ARGO_CONTAINER_ARGS_FILE is set and the referenced ConfigMap exists and is mounted at that exact path.","Verify the ConfigMap content is valid JSON of the container args (`kubectl get cm <pod-name> -o jsonpath='{.data}'`).","Upgrade workflow-controller, argo-server and executor images to the same Argo Workflows version so the args-offload contract matches.","Reduce the container args size (e.g. shorten script/argument payloads) so the controller does not take the offload path at all.","If it reproduces, check controller logs for ConfigMap creation errors around pod creation time."],"exampleFix":"// before: emissary started while ConfigMap volume was missing\n// spec.containers[*].volumeMounts missing the args configmap\n// after: ensure the offload configmap is mounted (controller does this automatically; custom patches must not remove it)\nvolumeMounts:\n  - name: argo-task-args        # configmap named after the pod\n    mountPath: /argo/staging    # must match ARGO_CONTAINER_ARGS_FILE dir","handlingStrategy":"validation","validationCode":"// Before submitting workflows with very large container args, verify the offload prerequisites:\nimport \"os\"\n\nfunc validateArgsOffloadReady() error {\n\tpath := os.Getenv(\"ARGO_CONTAINER_ARGS_FILE\")\n\tif path == \"\" {\n\t\treturn nil // offload not in use\n\t}\n\tf, err := os.Open(path)\n\tif err != nil {\n\t\treturn fmt.Errorf(\"args offload file %s not accessible: %w\", path, err)\n\t}\n\tdefer f.Close()\n\treturn nil\n}","typeGuard":null,"tryCatchPattern":"// On the error, surface the wrapped OS error for diagnosis:\nif err := runEmissary(ctx, name, includeScript, args); err != nil {\n\tif errors.Is(err, os.ErrNotExist) {\n\t\tlog.Errorf(\"offloaded args file missing — ConfigMap not mounted? %v\", err)\n\t} else if errors.Is(err, os.ErrPermission) {\n\t\tlog.Errorf(\"args offload file unreadable — check securityContext: %v\", err)\n\t}\n}","preventionTips":["Pin workflow-controller and executor images to the same version.","Never strip controller-injected env vars or volume mounts when patching pod specs.","Keep container args well under MaxEnvVarLen when possible to avoid the offload path.","Monitor namespace ConfigMaps for unexpected deletion (RBAC/GC policies)."],"tags":["kubernetes","executor","filesystem","env-var","argo-workflows"],"backgroundTag":"container-args-offload-file-unreadable","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T05:18:18.240Z"}