{"record":{"id":"4e6d360de4e63e8a","repo":"argoproj/argo-workflows","slug":"s-exceeded-its-deadline","errorCode":null,"errorMessage":"%s exceeded its deadline","messagePattern":"(.+?) exceeded its deadline","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"workflow/controller/workflowpod.go","lineNumber":955,"sourceCode":"\t// uses the same frozen clock as the rest of build, keeping build deterministic\n\t// for a given snapshot rather than leaking the live wall-clock in via deps.\n\tnode, err := pb.deps.getNodeByName(nodeName)\n\tif err != nil {\n\t\tlog.Warn(ctx, \"couldn't retrieve node, will get nil templateDeadline\")\n\t}\n\ttemplateDeadline, _, err := pb.deps.checkTemplateTimeouts(tmpl, node, pb.in.now)\n\tif err != nil {\n\t\treturn nil, err\n\t}\n\n\tif scheduleErr := pb.scheduleOnDifferentHost(ctx, node, tmpl, pod); scheduleErr != nil {\n\t\treturn nil, scheduleErr\n\t}\n\n\tif templateDeadline != nil && (pod.Spec.ActiveDeadlineSeconds == nil || pb.in.now.Sub(*templateDeadline).Seconds() < float64(*pod.Spec.ActiveDeadlineSeconds)) {\n\t\tnewActiveDeadlineSeconds := int64(templateDeadline.Sub(pb.in.now).Seconds())\n\t\tif newActiveDeadlineSeconds <= 1 {\n\t\t\treturn nil, fmt.Errorf(\"%s exceeded its deadline\", nodeName)\n\t\t}\n\t\tlog.WithFields(logging.Fields{\"newActiveDeadlineSeconds\": newActiveDeadlineSeconds, \"podNamespace\": pod.Namespace, \"podName\": pod.Name}).Debug(ctx, \"Setting new activeDeadlineSeconds\")\n\t\tpod.Spec.ActiveDeadlineSeconds = &newActiveDeadlineSeconds\n\t}\n\n\t// build is pure: the pod is fully constructed but not yet created. submitPod\n\t// performs the impure operations (ConfigMap/pod creation, rate limiting,\n\t// active-pod accounting, progress write).\n\tresult.Pod = pod\n\treturn result, nil\n}\n\nfunc (woc *wfOperationCtx) podExists(nodeID string) (existing *apiv1.Pod, exists bool, err error) {\n\tobjs, err := woc.controller.PodController.GetPodsByIndex(indexes.NodeIDIndex, woc.wf.Namespace+\"/\"+nodeID)\n\tif err != nil {\n\t\treturn nil, false, fmt.Errorf(\"failed to get pod from informer store: %w\", err)\n\t}\n","sourceCodeStart":937,"sourceCodeEnd":973,"githubUrl":"https://github.com/argoproj/argo-workflows/blob/35bff19146f5a6ada77468c431f2624bd577e373/workflow/controller/workflowpod.go#L937-L973","documentation":"When creating the pod, the controller computes a new ActiveDeadlineSeconds from the template's activeDeadline against the workflow/node deadline. If the remaining time is <= 1 second (deadline already effectively passed), it refuses to create the pod and returns this error instead.","triggerScenarios":"A step template with `activeDeadlineSeconds` is scheduled when the template's deadline (workflow deadline, step deadline, or retry node deadline) has already been reached or has less than ~1 second remaining at pod-build time.","commonSituations":"Long-queued workflows where earlier steps consumed the workflow's activeDeadlineSeconds; retry strategies whose accumulated deadline expired; a cron-suspended or delayed controller that reconciles the workflow after deadlines passed.","solutions":["Increase the workflow/template activeDeadlineSeconds or remove the deadline","Check why reconciliation was delayed (controller backlog, sync/semaphore holds) so steps start before deadlines","Restructure the workflow so deadline budgets match realistic step durations","If seen after controller downtime, delete/retry the node to reset its deadline state"],"exampleFix":"// before\nactiveDeadlineSeconds: 10  # earlier steps already consumed the budget\n// after\nactiveDeadlineSeconds: 300","handlingStrategy":"validation","validationCode":"// ensure workflow-level activeDeadlineSeconds exceeds the sum of step budgets\n// argo lint my-wf.yaml","typeGuard":null,"tryCatchPattern":"if newActiveDeadlineSeconds := int64(templateDeadline.Sub(now).Seconds()); newActiveDeadlineSeconds <= 1 {\n    return nil, fmt.Errorf(\"%s exceeded its deadline\", nodeName)\n}","preventionTips":["Budget activeDeadlineSeconds generously above worst-case step durations","Avoid tight deadlines when steps may queue behind semaphores/syncs","Monitor controller reconciliation lag","Use retryStrategy with its own backoff rather than ultra-short deadlines"],"tags":["deadline","timeout","scheduling"],"backgroundTag":"deadline-exceeded","analyzedSha":"35bff19146f5a6ada77468c431f2624bd577e373","analyzedAt":"2026-09-03T19:34:35.908Z","contentChangedAt":"2026-09-03T19:34:35.908Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}