{"record":{"id":"da3fda9237ccaa96","repo":"argoproj/argo-workflows","slug":"watching-supervisor-status-w","errorCode":null,"errorMessage":"watching supervisor status: %w","messagePattern":"watching supervisor status: %w","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"cmd/argoexec/commands/emissary.go","lineNumber":490,"sourceCode":"\tresCh := make(chan error, 1)\n\tfinish := func(err error) {\n\t\tselect {\n\t\tcase resCh <- err:\n\t\tdefault: // a result already landed; first one wins\n\t\t}\n\t\tcancel()\n\t}\n\n\t// Low-latency terminal detection: re-evaluate on every write to the marker\n\t// (heartbeats and the terminal write both fire here).\n\tgo func() {\n\t\twerr := file.WatchFile(watchCtx, statusPath, func() {\n\t\t\tif done, err := evaluateSupervisorStatus(statusPath, timeout, start); done {\n\t\t\t\tfinish(err)\n\t\t\t}\n\t\t})\n\t\tif werr != nil && watchCtx.Err() == nil {\n\t\t\tfinish(fmt.Errorf(\"watching supervisor status: %w\", werr))\n\t\t}\n\t}()\n\n\tticker := time.NewTicker(pollInterval)\n\tdefer ticker.Stop()\n\tfor {\n\t\tselect {\n\t\tcase <-ctx.Done():\n\t\t\treturn ctx.Err()\n\t\tcase err := <-resCh:\n\t\t\tif err == nil {\n\t\t\t\tlogger.Info(ctx, \"supervisor is ready\")\n\t\t\t}\n\t\t\treturn err\n\t\tcase <-ticker.C:\n\t\t\tif done, err := evaluateSupervisorStatus(statusPath, timeout, start); done {\n\t\t\t\tfinish(err)\n\t\t\t}","sourceCodeStart":472,"sourceCodeEnd":508,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/cmd/argoexec/commands/emissary.go#L472-L508","documentation":"In init-less pod mode, the emissary watches the supervisor's status marker with an inotify-based file watcher. If the watcher itself fails (its WatchFile call returns an error) while the watch context is still active, the emissary finishes the wait with this wrapped error rather than hanging.","triggerScenarios":"file.WatchFile returns an error while waiting for the supervisor status marker — e.g. the watched directory or file was removed, inotify limits were exhausted, or the watch could not be established on /var/run/argo/ctr/<name>.","commonSituations":"Host inotify watch limit exhausted (fs.inotify.max_user_watches); /var/run/argo directory cleaned up by another process; kernel/container environment lacking inotify support.","solutions":["Check the wrapped underlying error for the exact watcher failure","Raise the host inotify limits: sysctl fs.inotify.max_user_watches/max_user_instances (node-level, needs admin)","Ensure nothing in the pod deletes /var/run/argo/ctr/<name> while the workflow runs","Re-run the workflow; if transient it may succeed — if persistent, capture argoexec logs and node sysctl values"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"# On the node hosting workflows, verify inotify capacity:\nsysctl fs.inotify.max_user_watches fs.inotify.max_user_instances\n# Raise if exhausted (node-level):\nsudo sysctl -w fs.inotify.max_user_watches=524288","typeGuard":null,"tryCatchPattern":"if err := waitForSupervisorReady(ctx); err != nil {\n    if strings.Contains(err.Error(), \"watching supervisor status\") {\n        // transient watcher failure: retry once before failing the pod\n        time.Sleep(pollInterval)\n        return waitForSupervisorReady(ctx)\n    }\n    return err\n}","preventionTips":["Keep host inotify limits comfortably above the number of pods per node","Never delete or mutate /var/run/argo from sidecars or init containers","Use a kernel/container runtime with standard inotify support","Retry the workflow once for transient watcher errors before escalating"],"tags":["kubernetes","argo-workflows","argoexec","inotify","supervisor"],"backgroundTag":"inotify-watch-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}