conductor-oss/conductor · error · TerminateWorkflowException

Task %s/%s exceeded total timeout of %d seconds (elapsed %d

Error message

Task %s/%s exceeded total timeout of %d seconds (elapsed %d seconds across all attempts). No further retries will be attempted.

What it means

Guard inside the retry path: even if individual attempts would still retry, if TaskDef.totalTimeoutSeconds > 0 and the elapsed wall-clock since task.firstScheduledTime has reached that budget, the engine stops retrying and throws a TerminateWorkflowException (status TIMED_OUT if the last attempt was TIMED_OUT, else FAILED). totalTimeoutSeconds is a hard budget spanning all attempts AND retry delays.

Source

Thrown at core/src/main/java/com/netflix/conductor/core/execution/DeciderService.java:629

        if (taskDefinition.getTotalTimeoutSeconds() > 0 && task.getFirstScheduledTime() > 0) {
            long totalElapsedSeconds =
                    (System.currentTimeMillis() - task.getFirstScheduledTime()) / 1000;
            if (totalElapsedSeconds >= taskDefinition.getTotalTimeoutSeconds()) {
                final String errMsg =
                        String.format(
                                "Task %s/%s exceeded total timeout of %d seconds "
                                        + "(elapsed %d seconds across all attempts). "
                                        + "No further retries will be attempted.",
                                task.getTaskId(),
                                task.getTaskDefName(),
                                taskDefinition.getTotalTimeoutSeconds(),
                                totalElapsedSeconds);
                WorkflowModel.Status totalTimeoutStatus =
                        task.getStatus() == TaskModel.Status.TIMED_OUT
                                ? WorkflowModel.Status.TIMED_OUT
                                : WorkflowModel.Status.FAILED;
                updateWorkflowOutput(workflow, task);
                throw new TerminateWorkflowException(errMsg, totalTimeoutStatus, task);
            }
        }

        // retry... - but not immediately - put a delay...
        int startDelay = taskDefinition.getRetryDelaySeconds();
        switch (taskDefinition.getRetryLogic()) {
            case FIXED:
                startDelay = taskDefinition.getRetryDelaySeconds();
                startDelay = applyMaxRetryDelayCap(startDelay, taskDefinition);
                break;
            case LINEAR_BACKOFF:
                int linearRetryDelaySeconds =
                        taskDefinition.getRetryDelaySeconds()
                                * taskDefinition.getBackoffScaleFactor()
                                * (task.getRetryCount() + 1);
                // Reset integer overflow to max value
                startDelay =
                        linearRetryDelaySeconds < 0 ? Integer.MAX_VALUE : linearRetryDelaySeconds;

View on GitHub (pinned to cf7c3e4a8a)

Solutions

Raise TaskDef.totalTimeoutSeconds to cover attempts plus backoff, or set it to 0 to disable the total budget.
Tune retryDelaySeconds and backoffScaleFactor so cumulative delay fits the budget.
Fix the underlying task so it succeeds before the budget elapses.
Switch retryLogic to a faster policy if the budget is tight.

Defensive patterns

Strategy: validation

Validate before calling

// Warn if total budget < sum of (max attempts * typical duration + backoff).
int attempts = td.getRetryCount() + 1;
long est = attempts * typicalAttemptSeconds + estimatedBackoffSeconds(td);
if (td.getTotalTimeoutSeconds() > 0 && est > td.getTotalTimeoutSeconds()) {
    LOGGER.warn("totalTimeoutSeconds {} likely too low; estimated {}", td.getTotalTimeoutSeconds(), est);
}

Try / catch

try {
    deciderService.decide(workflow);
} catch (TerminateWorkflowException e) {
    if (e.getMessage().contains("exceeded total timeout")) {
        // raise budget or reduce backoff
    }
}

Prevention

Size totalTimeoutSeconds to cover attempts AND retry delays.
Cap retryDelaySeconds / backoffScaleFactor so cumulative delay fits the budget.
Set totalTimeoutSeconds=0 to disable the total budget if undesired.

When it happens

Trigger: A task with totalTimeoutSeconds configured has, across all its attempts and backoff delays, consumed more wall-clock than the configured total budget, and another retry is being considered.

Common situations: LINEAR/EXPONENTIAL backoff delays accumulate past the total budget; retries keep failing slowly so the sum of attempt+durations crosses totalTimeoutSeconds; totalTimeoutSeconds set lower than realistic end-to-end time.

Understand the failure class

Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

AI-assisted analysis of conductor-oss/conductor@cf7c3e4a8a (2026-08-14). Data as JSON: /api/errors/b72380977100cfc9. Report an issue: GitHub.