{"record":{"id":"09491c6c1631e0a6","repo":"apache/cassandra","slug":"truncate-failed-on-replica-endpoint-failurer","errorCode":null,"errorMessage":"Truncate failed on replica {endpoint} -> {failureReason}","messagePattern":"Truncate failed on replica (.+?) -> (.+?)","errorType":"exception","errorClass":"TruncateException","httpStatus":null,"severity":"error","filePath":"src/java/org/apache/cassandra/service/TruncateResponseHandler.java","lineNumber":90,"sourceCode":"        {\n            throw new UncheckedInterruptedException(e);\n        }\n\n        if (!signaled)\n            throw new TimeoutException(\"Truncate timed out - received only \" + responses.get() + \" responses\");\n\n        if (!failureReasonByEndpoint.isEmpty())\n        {\n            // clone to make sure no race condition happens\n            Map<InetAddressAndPort, RequestFailureReason> failureReasonByEndpoint = new HashMap<>(this.failureReasonByEndpoint);\n            if (RequestCallback.isTimeout(failureReasonByEndpoint))\n                throw new TimeoutException(\"Truncate timed out - received only \" + responses.get() + \" responses\");\n\n            StringBuilder sb = new StringBuilder(\"Truncate failed on \");\n            for (Map.Entry<InetAddressAndPort, RequestFailureReason> e : failureReasonByEndpoint.entrySet())\n                sb.append(\"replica \").append(e.getKey()).append(\" -> \").append(e.getValue()).append(\", \");\n            sb.setLength(sb.length() - 2);\n            throw new TruncateException(sb.toString());\n        }\n    }\n\n    @Override\n    public void onResponse(Message<TruncateResponse> message)\n    {\n        responses.incrementAndGet();\n        if (responses.get() >= responseCount)\n            condition.signalAll();\n    }\n\n    @Override\n    public void onFailure(InetAddressAndPort from, RequestFailure failure)\n    {\n        // If the truncation hasn't succeeded on some replica, abort and indicate this back to the client.\n        failureReasonByEndpoint.put(from, failure.reason);\n        condition.signalAll();\n    }","sourceCodeStart":72,"sourceCodeEnd":108,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/service/TruncateResponseHandler.java#L72-L108","documentation":"Thrown by TruncateResponseHandler.get() as TruncateException when at least one replica explicitly reported a non-timeout failure reason for the truncation. The message lists each failing replica and its RequestFailureReason so the operator knows which nodes failed and why.","triggerScenarios":"TRUNCATE where a replica sends a failure response with a concrete RequestFailureReason (e.g. UNKNOWN, node crashing during truncation, replica-local TruncateException) back to the coordinator.","commonSituations":"A replica crashed or restarted mid-truncation, disk full on a replica, replica threw while dropping SSTables/memtables, or mixed-version cluster where a replica cannot process the truncate verb.","solutions":["Read the replica endpoints in the message and check those nodes' logs for the underlying truncation error.","Fix the replica-side issue (disk space, restart, repair) and retry the TRUNCATE.","If a replica is persistently unhealthy, replace it (nodetool decommission/rebuild) and re-run truncate.","Check for version incompatibilities if failure reasons look protocol-related after an upgrade."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// check disk space and liveness of each replica before truncating\nreplicas.forEach(r -> assertReplicaHealthy(r, minFreeDisk, truncateRequestTimeoutMs));","typeGuard":null,"tryCatchPattern":"catch (TruncateException e) {\n    Set<InetAddressAndPort> failed = parseFailedReplicas(e.getMessage());\n    failed.forEach(r -> alertOps(\"Replica failed truncate\", r));\n    throw e; // do not assume data was removed\n}","preventionTips":["Alert on per-replica disk space and error rates before running DDL-like operations.","Treat TruncateException as 'data possibly still present on listed replicas' and verify.","Keep cluster versions homogeneous during rolling upgrades before truncating."],"tags":["truncate","replica-failure","distributed","storage"],"backgroundTag":"replica-failure","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}