{"record":{"id":"7d1143d17e2c17f0","repo":"apache/cassandra","slug":"request-has-spent-over-s-time-of-the-maximum-time","errorCode":null,"errorMessage":"Request has spent over %s time of the maximum timeout %dms in the queue","messagePattern":"Request has spent over (.+?) time of the maximum timeout (.+?)ms in the queue","errorType":"console","errorClass":null,"httpStatus":null,"severity":"info","filePath":"src/java/org/apache/cassandra/transport/Dispatcher.java","lineNumber":423,"sourceCode":"                break;\n            }\n            case BYTES_IN_FLIGHT:\n            {\n                String message = String.format(\"Request breached limit(s) on bytes in flight (Endpoint: %d, Global: %d) and triggered backpressure.\",\n                                               ClientResourceLimits.getEndpointLimit(), ClientResourceLimits.getGlobalLimit());\n\n                NoSpamLogger.log(logger, NoSpamLogger.Level.INFO, 1, TimeUnit.MINUTES, message);\n                ClientWarn.instance.warn(message);\n                break;\n            }\n            case QUEUE_TIME:\n            {\n                String message = String.format(\"Request has spent over %s time of the maximum timeout %dms in the queue\",\n                                               DatabaseDescriptor.getNativeTransportQueueMaxItemAgeThreshold(),\n                                               DatabaseDescriptor.getNativeTransportTimeout(TimeUnit.MILLISECONDS));\n\n                NoSpamLogger.log(logger, NoSpamLogger.Level.INFO, 1, TimeUnit.MINUTES, message);\n                ClientWarn.instance.warn(message);\n                break;\n            }\n        }\n\n        QueryState qstate = connection.validateNewMessage(request.type, connection.getVersion());\n\n        Message.logger.trace(\"Received: {}, v={}\", request, connection.getVersion());\n        Message.Response response = request.execute(qstate, requestTime);\n\n        if (request.isTrackable())\n        {\n            CoordinatorWarnings.done();\n            CoordinatorWriteWarnings.done();\n        }\n\n        response.attach(connection);\n        connection.applyStateTransition(request.type, response.type);\n        return response;","sourceCodeStart":405,"sourceCodeEnd":441,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/transport/Dispatcher.java#L405-L441","documentation":"A backpressure notice from the Dispatcher when a request has already waited longer in the coordination queue than native_transport_queue_max_item_age_threshold relative to the native transport timeout. Logged INFO and delivered as a client warning; it flags server-side queuing saturation rather than a failed request.","triggerScenarios":"native_transport_backpressure=QUEUE_TIME and the elapsed queue time exceeds DatabaseDescriptor.getNativeTransportQueueMaxItemAgeThreshold() (a fraction of getNativeTransportTimeout) when the request is finally dequeued in processRequest.","commonSituations":"Request bursts exceeding coordinator throughput; too few coordinator threads (native_transport_max_threads); a slow endpoint backing up the queue during repair/streaming; long GC pauses delaying processing.","solutions":["Increase native_transport_max_threads if CPU headroom exists so the queue drains faster.","Tune native_transport_timeout and native_transport_queue_max_item_age_threshold to values that suit your latency envelope.","Find and fix slow operations (repairs, huge reads, dropped mutations) that are saturating the coordinator.","Smooth client traffic (rate limiting/backoff) to avoid queue buildup during bursts."],"exampleFix":"// cassandra.yaml: before native_transport_timeout: 12000ms // after (larger queue tolerance) native_transport_timeout: 30000ms","handlingStrategy":"retry","validationCode":"// measure coordinator latency before burst submission: if (recentP99QueueTimeMs > timeoutBudgetMs) throttleSubmissionRate();","typeGuard":null,"tryCatchPattern":"if (executionInfo.getWarnings() != null && executionInfo.getWarnings().stream().anyMatch(w -> w.contains(\"in the queue\"))) { retryWithBackoff(request); }","preventionTips":["Right-size native_transport_max_threads for the workload.","Monitor coordinator queue depths and GC pauses.","Avoid scheduling repairs/streaming during peak client traffic.","Set queue_max_item_age_threshold consistent with client timeout budgets."],"tags":["native-transport","queue-time","latency","backpressure"],"backgroundTag":"request-timeout","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}