{"record":{"id":"c1f3017cf2b0355a","repo":"apache/cassandra","slug":"failure-for-repair-verb-could-not-complete","errorCode":null,"errorMessage":"{} {} failure for repair verb ; could not complete within {} attempts","messagePattern":"(.+?) (.+?) failure for repair verb ; could not complete within (.+?) attempts","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/repair/messages/RepairMessage.java","lineNumber":198,"sourceCode":"            throw new AssertionError(\"Repair verb \" + verb + \" does not support retry, but a request to send with retry was given!\");\n        BiConsumer<Integer, RequestFailureReason > maybeRecordRetry = (attempt, reason) -> {\n            if (attempt <= 1)\n                return;\n            // we don't know what the prefix kind is... so use NONE... this impacts logPrefix as it will cause us to use \"repair\" rather than \"preview repair\" which may not be correct... but close enough...\n            String prefix = PreviewKind.NONE.logPrefix(request.parentRepairSession());\n            RepairMetrics.retry(verb, attempt);\n            if (reason == null)\n            {\n                noSpam.info(\"{} Retry of repair verb \" + verb + \" was successful after {} attempts\", prefix, attempt);\n            }\n            else if (reason == RequestFailureReason.TIMEOUT)\n            {\n                noSpam.warn(\"{} Timeout for repair verb \" + verb + \"; could not complete within {} attempts\", prefix, attempt);\n                RepairMetrics.retryTimeout(verb);\n            }\n            else\n            {\n                noSpam.warn(\"{} {} failure for repair verb \" + verb + \"; could not complete within {} attempts\", prefix, reason, attempt);\n                RepairMetrics.retryFailure(verb);\n            }\n        };\n        ctx.messaging().sendWithRetries(backoff, ctx.optionalTasks()::schedule,\n                                        verb, request, Iterators.cycle(endpoint),\n                                        (int attempt, Message<T> msg, Throwable failure) -> {\n                                            if (failure == null)\n                                            {\n                                                maybeRecordRetry.accept(attempt, null);\n                                                finalCallback.onResponse(msg);\n                                            }\n                                        },\n                                        (attempt, from, failure) -> {\n                                            ErrorHandling allowed = errorHandlingSupported(ctx, endpoint, verb, request.parentRepairSession());\n                                            switch (allowed)\n                                            {\n                                                case NONE:\n                                                    logger.error(\"[#{}] {} failed on {}: {}\", request.parentRepairSession(), verb, from, failure);","sourceCodeStart":180,"sourceCodeEnd":216,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/repair/messages/RepairMessage.java#L180-L216","documentation":"This is a warn-level log emitted by RepairMessage.sendMessageWithRetries when a repair message could not be delivered/replied to within the configured retry attempts due to a failure (as opposed to a timeout). It reports the failure reason and the number of attempts exhausted, and increments RepairMetrics.retryFailure for the verb.","triggerScenarios":"Calling sendMessageWithRetries for a repair verb where the messaging layer's sendWithRetries exhausts all retry attempts with a non-timeout failure (e.g. node down, connection dropped, response failure) at the configured backoff limit.","commonSituations":"Repairing a cluster with an unreachable or restarting node; network partitions during repair; a target endpoint failing to respond to repair messages; messaging-layer failures (TLS handshake, dropped connections) consuming all retries.","solutions":["Check the referenced node's logs and connectivity (nodetool status, ping) and restart the repair once the node is healthy","Increase repair message retry settings (repair message timeout/retry configuration) if failures are transient","Upgrade the peer node to a version supporting repair message retries (>= SUPPORTS_RETRY version) so failures are handled with retry/timeout semantics","Tune backoff/attempt counts in the messaging retry backoff configuration"],"exampleFix":"// before\nctx.messaging().sendWithRetries(backoff, ...); // small backoff, node transiently down\n// after\n// fix node connectivity first, then retry repair\nnodetool repair -- myks mytable; // after confirming 'nodetool status' shows the peer UL/UN","handlingStrategy":"retry","validationCode":"// before repair\n// verify all endpoints are alive and reachable\nfor (InetAddressAndPort ep : endpoints)\n    if (!FailureDetector.instance.isAlive(ep))\n        throw new IllegalStateException(\"Endpoint down before repair send: \" + ep);","typeGuard":null,"tryCatchPattern":"// retries are internal; surface exhaustion\ntry {\n    sendMessageWithRetries(...);\n} catch (RepairException | RuntimeException e) {\n    logger.error(\"Repair verb {} failed after retries\", verb, e);\n    RepairMetrics.retryFailure(verb);\n}","preventionTips":["Monitor RepairMetrics retry/failure counters for recurring node issues","Keep repair message timeout/retry settings adequate for cluster size","Fix node liveness/network issues before initiating repairs","Upgrade peers to versions supporting repair retries"],"tags":["repair","messaging","retry-exhausted","network"],"backgroundTag":"request-timeout","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}