{"record":{"id":"b202f9613d251bcf","repo":"argoproj/argo-workflows","slug":"failed-to-wait-for-main-container-to-complete-w","errorCode":null,"errorMessage":"failed to wait for main container to complete: %w","messagePattern":"failed to wait for main container to complete: %w","errorType":"error_code","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/executor/executor.go","lineNumber":1378,"sourceCode":"\t// only monitor progress if both tick durations are >0\n\tif we.annotationPatchTickDuration != 0 && we.readProgressFileTickDuration != 0 {\n\t\tgo we.monitorProgress(ctx, we.progressFile)\n\t} else {\n\t\tlogger.WithField(\"annotationPatchTickDuration\", we.annotationPatchTickDuration).WithField(\"readProgressFileTickDuration\", we.readProgressFileTickDuration).Info(ctx, \"monitoring progress disabled\")\n\t}\n\n\tgo we.monitorDeadline(ctx, containerNames)\n\n\terr := retryutil.OnError(we.retryBackoff, func(err error) bool {\n\t\treturn errorsutil.IsTransientErr(ctx, err)\n\t}, func() error {\n\t\treturn we.waitMainContainers(ctx, containerNames)\n\t})\n\n\tlogger.WithError(err).Info(ctx, \"Main container completed\")\n\n\tif err != nil && !errors.Is(err, context.Canceled) {\n\t\treturn fmt.Errorf(\"failed to wait for main container to complete: %w\", err)\n\t}\n\treturn nil\n}\n\n// waitMainContainers blocks until the given containers have completed, as\n// signalled by the emissary's per-container exit-code files.\nfunc (we *WorkflowExecutor) waitMainContainers(ctx context.Context, containerNames []string) error {\n\treturn we.RuntimeExecutor.Wait(ctx, containerNames)\n}\n\n// monitorProgress monitors for self-reported progress in the progressFile and patches the pod annotations with the parsed progress.\n//\n// The function watches the `progressFile` via inotify and re-parses the last line on every write.\n// If the line matches `N/M`, will set the progress annotation to the parsed progress value.\n// Every `annotationPatchTickDuration` the pod is patched with the updated annotations. This way the controller\n// gets notified of new self reported progress.\nfunc (we *WorkflowExecutor) monitorProgress(ctx context.Context, progressFile string) {\n\tlogger := logging.RequireLoggerFromContext(ctx)","sourceCodeStart":1360,"sourceCodeEnd":1396,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/executor/executor.go#L1360-L1396","documentation":"Wraps any failure from waitMainContainers in the emissary executor: the executor could not observe the main container(s) completing successfully while waiting on per-container exit-code files under /var/run/argo/ctr. The wrapped error tells why the wait failed (file never appears, read error, abnormal completion detection). context.Canceled is deliberately not wrapped, since cancellation is an intentional shutdown.","triggerScenarios":"During PostMain/`argoexec emissary` output capture, the exit-code file for a main container is missing or unreadable within the deadline, or the underlying wait reports an error other than context cancellation.","commonSituations":"Main container OOM-killed or node deleted so the exit-code file is never written; disk pressure wiping /var/run/argo; pod terminating mid-capture so the wait context is cancelled (this one is filtered, so seeing the wrapped error means non-cancel failure); race between emissary supervisor and wait stage on container teardown.","solutions":["Read the wrapped error (`%w` cause) to identify the specific failure — e.g. missing exit-code file vs read error","Check the main container's status: `kubectl describe pod <pod>` and `kubectl logs <pod> -c main` for OOMKilled/eviction","Verify /var/run/argo/ctr/<container>/exitcode exists and the emissary supervisor wrote it; check argoexec logs (`argo logs <workflow>`)","Re-run the workflow; if reproducible on a node, cordon/inspect the node for disk or kubelet issues","If it happens only at workflow shutdown, confirm the executor version matches (older versions had wait/teardown races)"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err != nil && !errors.Is(err, context.Canceled) {\n    var mainErr error\n    if errors.As(err, &mainErr) {\n        log.Printf(\"main container wait failed: %v\", mainErr) // inspect wrapped cause\n    }\n}\n// then check the pod: kubectl describe pod & logs -c main for OOMKilled/eviction","preventionTips":["Always read the wrapped cause to distinguish missing exit-code file from read errors","Check the main container for OOMKilled/evicted events when this appears","Ensure /var/run/argo is on a writable, non-pressure volume","Keep executor and controller versions in sync"],"tags":["executor","argoexec","containers","pod"],"backgroundTag":"main-container-wait-failed","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}