argoproj/argo-workflows · error

failed to wait for main container to complete: %w

Error message

failed to wait for main container to complete: %w

What it means

Wraps any failure from waitMainContainers in the emissary executor: the executor could not observe the main container(s) completing successfully while waiting on per-container exit-code files under /var/run/argo/ctr. The wrapped error tells why the wait failed (file never appears, read error, abnormal completion detection). context.Canceled is deliberately not wrapped, since cancellation is an intentional shutdown.

Source

Thrown at workflow/executor/executor.go:1378

	// only monitor progress if both tick durations are >0
	if we.annotationPatchTickDuration != 0 && we.readProgressFileTickDuration != 0 {
		go we.monitorProgress(ctx, we.progressFile)
	} else {
		logger.WithField("annotationPatchTickDuration", we.annotationPatchTickDuration).WithField("readProgressFileTickDuration", we.readProgressFileTickDuration).Info(ctx, "monitoring progress disabled")
	}

	go we.monitorDeadline(ctx, containerNames)

	err := retryutil.OnError(we.retryBackoff, func(err error) bool {
		return errorsutil.IsTransientErr(ctx, err)
	}, func() error {
		return we.waitMainContainers(ctx, containerNames)
	})

	logger.WithError(err).Info(ctx, "Main container completed")

	if err != nil && !errors.Is(err, context.Canceled) {
		return fmt.Errorf("failed to wait for main container to complete: %w", err)
	}
	return nil
}

// waitMainContainers blocks until the given containers have completed, as
// signalled by the emissary's per-container exit-code files.
func (we *WorkflowExecutor) waitMainContainers(ctx context.Context, containerNames []string) error {
	return we.RuntimeExecutor.Wait(ctx, containerNames)
}

// monitorProgress monitors for self-reported progress in the progressFile and patches the pod annotations with the parsed progress.
//
// The function watches the `progressFile` via inotify and re-parses the last line on every write.
// If the line matches `N/M`, will set the progress annotation to the parsed progress value.
// Every `annotationPatchTickDuration` the pod is patched with the updated annotations. This way the controller
// gets notified of new self reported progress.
func (we *WorkflowExecutor) monitorProgress(ctx context.Context, progressFile string) {
	logger := logging.RequireLoggerFromContext(ctx)

View on GitHub (pinned to 35bff19146)

Solutions

1. Read the wrapped error (`%w` cause) to identify the specific failure — e.g. missing exit-code file vs read error
2. Check the main container's status: `kubectl describe pod <pod>` and `kubectl logs <pod> -c main` for OOMKilled/eviction
3. Verify /var/run/argo/ctr/<container>/exitcode exists and the emissary supervisor wrote it; check argoexec logs (`argo logs <workflow>`)
4. Re-run the workflow; if reproducible on a node, cordon/inspect the node for disk or kubelet issues
5. If it happens only at workflow shutdown, confirm the executor version matches (older versions had wait/teardown races)

if err != nil && !errors.Is(err, context.Canceled) {
    var mainErr error
    if errors.As(err, &mainErr) {
        log.Printf("main container wait failed: %v", mainErr) // inspect wrapped cause
    }
}
// then check the pod: kubectl describe pod & logs -c main for OOMKilled/eviction

When it happens

Trigger: During PostMain/`argoexec emissary` output capture, the exit-code file for a main container is missing or unreadable within the deadline, or the underlying wait reports an error other than context cancellation.

Common situations: Main container OOM-killed or node deleted so the exit-code file is never written; disk pressure wiping /var/run/argo; pod terminating mid-capture so the wait context is cancelled (this one is filtered, so seeing the wrapped error means non-cancel failure); race between emissary supervisor and wait stage on container teardown.

Related errors

• not supported
• %w
• %w
• failed to create ctr directory: %w
• failed to read container args file %s: %w

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/b202f9613d251bcf. Report an issue: GitHub.