apache/cassandra · info

Request has spent over

Error message

Request has spent over %s time of the maximum timeout %dms in the queue

What it means

A backpressure notice from the Dispatcher when a request has already waited longer in the coordination queue than native_transport_queue_max_item_age_threshold relative to the native transport timeout. Logged INFO and delivered as a client warning; it flags server-side queuing saturation rather than a failed request.

Solutions

1. Increase native_transport_max_threads if CPU headroom exists so the queue drains faster.
2. Tune native_transport_timeout and native_transport_queue_max_item_age_threshold to values that suit your latency envelope.
3. Find and fix slow operations (repairs, huge reads, dropped mutations) that are saturating the coordinator.
4. Smooth client traffic (rate limiting/backoff) to avoid queue buildup during bursts.

Example fix

// cassandra.yaml: before native_transport_timeout: 12000ms // after (larger queue tolerance) native_transport_timeout: 30000ms

// measure coordinator latency before burst submission: if (recentP99QueueTimeMs > timeoutBudgetMs) throttleSubmissionRate();

if (executionInfo.getWarnings() != null && executionInfo.getWarnings().stream().anyMatch(w -> w.contains("in the queue"))) { retryWithBackoff(request); }

When it happens

Trigger: native_transport_backpressure=QUEUE_TIME and the elapsed queue time exceeds DatabaseDescriptor.getNativeTransportQueueMaxItemAgeThreshold() (a fraction of getNativeTransportTimeout) when the request is finally dequeued in processRequest.

Common situations: Request bursts exceeding coordinator throughput; too few coordinator threads (native_transport_max_threads); a slow endpoint backing up the queue during repair/streaming; long GC pauses delaying processing.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Request breached global limit of
• Request breached limit(s) on bytes in flight
• Artificial latency limit is
• be positive
• Disabling gossip while native transport is still active is…

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/7d1143d17e2c17f0. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/transport/Dispatcher.java:423

                break;
            }
            case BYTES_IN_FLIGHT:
            {
                String message = String.format("Request breached limit(s) on bytes in flight (Endpoint: %d, Global: %d) and triggered backpressure.",
                                               ClientResourceLimits.getEndpointLimit(), ClientResourceLimits.getGlobalLimit());

                NoSpamLogger.log(logger, NoSpamLogger.Level.INFO, 1, TimeUnit.MINUTES, message);
                ClientWarn.instance.warn(message);
                break;
            }
            case QUEUE_TIME:
            {
                String message = String.format("Request has spent over %s time of the maximum timeout %dms in the queue",
                                               DatabaseDescriptor.getNativeTransportQueueMaxItemAgeThreshold(),
                                               DatabaseDescriptor.getNativeTransportTimeout(TimeUnit.MILLISECONDS));

                NoSpamLogger.log(logger, NoSpamLogger.Level.INFO, 1, TimeUnit.MINUTES, message);
                ClientWarn.instance.warn(message);
                break;
            }
        }

        QueryState qstate = connection.validateNewMessage(request.type, connection.getVersion());

        Message.logger.trace("Received: {}, v={}", request, connection.getVersion());
        Message.Response response = request.execute(qstate, requestTime);

        if (request.isTrackable())
        {
            CoordinatorWarnings.done();
            CoordinatorWriteWarnings.done();
        }

        response.attach(connection);
        connection.applyStateTransition(request.type, response.type);
        return response;

View on GitHub (pinned to 88fd0f6a0e)