argoproj/argo-workflows · error

%s exceeded its deadline

Error message

%s exceeded its deadline

What it means

When creating the pod, the controller computes a new ActiveDeadlineSeconds from the template's activeDeadline against the workflow/node deadline. If the remaining time is <= 1 second (deadline already effectively passed), it refuses to create the pod and returns this error instead.

Source

Thrown at workflow/controller/workflowpod.go:955

	// uses the same frozen clock as the rest of build, keeping build deterministic
	// for a given snapshot rather than leaking the live wall-clock in via deps.
	node, err := pb.deps.getNodeByName(nodeName)
	if err != nil {
		log.Warn(ctx, "couldn't retrieve node, will get nil templateDeadline")
	}
	templateDeadline, _, err := pb.deps.checkTemplateTimeouts(tmpl, node, pb.in.now)
	if err != nil {
		return nil, err
	}

	if scheduleErr := pb.scheduleOnDifferentHost(ctx, node, tmpl, pod); scheduleErr != nil {
		return nil, scheduleErr
	}

	if templateDeadline != nil && (pod.Spec.ActiveDeadlineSeconds == nil || pb.in.now.Sub(*templateDeadline).Seconds() < float64(*pod.Spec.ActiveDeadlineSeconds)) {
		newActiveDeadlineSeconds := int64(templateDeadline.Sub(pb.in.now).Seconds())
		if newActiveDeadlineSeconds <= 1 {
			return nil, fmt.Errorf("%s exceeded its deadline", nodeName)
		}
		log.WithFields(logging.Fields{"newActiveDeadlineSeconds": newActiveDeadlineSeconds, "podNamespace": pod.Namespace, "podName": pod.Name}).Debug(ctx, "Setting new activeDeadlineSeconds")
		pod.Spec.ActiveDeadlineSeconds = &newActiveDeadlineSeconds
	}

	// build is pure: the pod is fully constructed but not yet created. submitPod
	// performs the impure operations (ConfigMap/pod creation, rate limiting,
	// active-pod accounting, progress write).
	result.Pod = pod
	return result, nil
}

func (woc *wfOperationCtx) podExists(nodeID string) (existing *apiv1.Pod, exists bool, err error) {
	objs, err := woc.controller.PodController.GetPodsByIndex(indexes.NodeIDIndex, woc.wf.Namespace+"/"+nodeID)
	if err != nil {
		return nil, false, fmt.Errorf("failed to get pod from informer store: %w", err)
	}

View on GitHub (pinned to 35bff19146)

Solutions

1. Increase the workflow/template activeDeadlineSeconds or remove the deadline
2. Check why reconciliation was delayed (controller backlog, sync/semaphore holds) so steps start before deadlines
3. Restructure the workflow so deadline budgets match realistic step durations
4. If seen after controller downtime, delete/retry the node to reset its deadline state

Example fix

// before
activeDeadlineSeconds: 10  # earlier steps already consumed the budget
// after
activeDeadlineSeconds: 300

// ensure workflow-level activeDeadlineSeconds exceeds the sum of step budgets
// argo lint my-wf.yaml

if newActiveDeadlineSeconds := int64(templateDeadline.Sub(now).Seconds()); newActiveDeadlineSeconds <= 1 {
    return nil, fmt.Errorf("%s exceeded its deadline", nodeName)
}

When it happens

Trigger: A step template with `activeDeadlineSeconds` is scheduled when the template's deadline (workflow deadline, step deadline, or retry node deadline) has already been reached or has less than ~1 second remaining at pod-build time.

Common situations: Long-queued workflows where earlier steps consumed the workflow's activeDeadlineSeconds; retry strategies whose accumulated deadline expired; a cron-suspended or delayed controller that reconciles the workflow after deadlines passed.

Related errors

• supervisor presumed dead: status marker never appeared withi
• %w: %w
• plugin %s context cancelled while waiting for socket at %q:
• plugin %s expected unix socket at %q but it does not exist a
• timeout waiting for plugin %s connection to become ready, la

AI-assisted analysis of argoproj/argo-workflows@35bff19146 (2026-09-03). Data as JSON: /api/errors/4e6d360de4e63e8a. Report an issue: GitHub.