conductor-oss/conductor · error · TransientException

Failed to check rate limit for task: %s in taskDef: %s

Error message

Failed to check rate limit for task: %s in taskDef: %s

What it means

Thrown when a DriverException occurs inside exceedsRateLimitPerFrequency while counting the current rate-limit bucket (SELECT count) or recording a new execution (INSERT timeuuid). Wrapped as TransientException - the rate-limit check could not complete due to an infrastructure failure. Note the rate limit itself is approximate: count and insert are separate statements, so concurrent pollers can overshoot.

Source

Thrown at cassandra-persistence/src/main/java/com/netflix/conductor/cassandra/dao/CassandraRateLimitingDAO.java:134

                return false;
            } else {
                LOGGER.info(
                        "TaskId: {} with TaskDefinition of: {} has rateLimitPerFrequency: {} and rateLimitFrequencyInSeconds: {} is out of bounds of rate limit with current count {}",
                        task.getTaskId(),
                        task.getTaskDefName(),
                        rateLimitPerFrequency,
                        rateLimitFrequencyInSeconds,
                        currentBucketCount);
                return true;
            }
        } catch (DriverException e) {
            Monitors.error(CLASS_NAME, "exceedsRateLimitPerFrequency");
            String errorMsg =
                    String.format(
                            "Failed to check rate limit for task: %s in taskDef: %s",
                            task.getTaskId(), task.getTaskDefName());
            LOGGER.error(errorMsg, e);
            throw new TransientException(errorMsg, e);
        }
    }
}

View on GitHub (pinned to cf7c3e4a8a)

Solutions

1. Retry the rate-limit check with backoff - it is a TransientException.
2. Verify the task_rate_limit table exists (run the schema migration).
3. Ensure conductor.cassandra.readConsistencyLevel is at least quorum on multi-node clusters to avoid stale window counts.
4. Check the wrapped DriverException for timeout vs unavailable.

Example fix

// before: single check, surfaces TransientException
boolean limited = rateLimitingDAO.exceedsRateLimitPerFrequency(task, taskDef);

// after: retry transient failures
boolean limited = RetryUtils.retryOn(TransientException.class, 3,
        Duration.ofMillis(100),
        () -> rateLimitingDAO.exceedsRateLimitPerFrequency(task, taskDef));

// Pre-flight: confirm session before rate-limit check
if (cassandraSession.isClosed()) {
    throw new IllegalStateException("Cassandra session is closed; cannot check rate limit");
}

// Retry transient rate-limit checks; on persistent failure, fail safe per policy
try {
    return rateLimitingDAO.exceedsRateLimitPerFrequency(task, taskDef);
} catch (TransientException e) {
    return backoffAndRetry(
            () -> rateLimitingDAO.exceedsRateLimitPerFrequency(task, taskDef), 3);
}

When it happens

Trigger: session.execute(selectRateLimitCountStatement.bind(...)) or session.execute(insertRateLimitBucketStatement.bind(...)) raises a DriverException during the per-frequency rate-limit evaluation for a task.

Common situations: Cassandra degradation during high poll throughput; task_rate_limit table missing (schema not migrated); read consistency below quorum on a multi-node cluster returning a stale window; connection pool exhaustion.

Related errors

• Failed to get workflow: %s
• Error creating workflow definition: %s/%d
• Error updating workflow definition: %s/%d
• Error fetching workflow def: %s/%d
• Failed to remove workflow definition: %s/%d

AI-assisted analysis of conductor-oss/conductor@cf7c3e4a8a (2026-08-14). Data as JSON: /api/errors/ba86b62d71fee304. Report an issue: GitHub.