{"record":{"id":"02ac4af6756320dc","repo":"apache/cassandra","slug":"encountered-d-unexpected-exceptions-while-sending","errorCode":null,"errorMessage":"Encountered %d unexpected exceptions while sending out batches","messagePattern":"Encountered (.+?) unexpected exceptions while sending out batches","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"src/java/org/apache/cassandra/batchlog/BatchlogManager.java","lineNumber":337,"sourceCode":"            }\n\n            if (++positionInPage == pageSize)\n            {\n                // We have reached the end of a batch. To avoid keeping more than a page of mutations in memory,\n                // finish processing the page before requesting the next row.\n                finishAndClearBatches(unfinishedBatches, hintedNodes, replayedBatches);\n                positionInPage = 0;\n            }\n        }\n\n        // finalize the incomplete last page of batches\n        if (positionInPage > 0)\n            finishAndClearBatches(unfinishedBatches, hintedNodes, replayedBatches);\n        else\n            logger.trace(\"Had no batches to replay\");\n\n        if (caughtException != null)\n            logger.warn(String.format(\"Encountered %d unexpected exceptions while sending out batches\", skipped), caughtException);\n\n        // to preserve batch guarantees, we must ensure that hints (if any) have made it to disk, before deleting the batches\n        HintsService.instance.flushAndFsyncBlockingly(hintedNodes);\n\n        // once all generated hints are fsynced, actually delete the batches\n        replayedBatches.forEach(BatchlogManager::remove);\n    }\n\n    private void dispatchBatch(RateLimiter rateLimiter, Row row, TimeUUID id, int version, Set<UUID> hintedNodes, ArrayList<ReplayingBatch> unfinishedBatches) throws IOException\n    {\n        while (true)\n        {\n            ClusterMetadata cm = ClusterMetadata.current();\n            try\n            {\n                ReplayingBatch batch = new ReplayingBatch(id, version, row.getList(\"mutations\", BytesType.instance), cm);\n                if (batch.replay(rateLimiter, hintedNodes))\n                {","sourceCodeStart":319,"sourceCodeEnd":355,"githubUrl":"https://github.com/apache/cassandra/blob/88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1/src/java/org/apache/cassandra/batchlog/BatchlogManager.java#L319-L355","documentation":"At the end of processBatchlogEntries, if any batches were skipped during the replay pass (each logged individually), BatchlogManager logs this aggregated warning with the skip count plus the retained exception. It signals that batchlog guarantees could not be fully honored for `skipped` batches; hints for other batches are still flushed and fsynced before those batches are deleted.","triggerScenarios":"One or more dispatchBatch calls threw IOException during replayFailedBatches, incrementing the skipped counter — corrupt batchlog payloads, deserialization failures, or I/O errors.","commonSituations":"Clusters recovering from disk corruption or unclean shutdowns; replay after upgrades where stored batches were written with incompatible versions; repeated warnings across replays indicate persistent corruption in system.batches.","solutions":["Read the attached cause (caughtException) and the per-batch 'Skipped batch replay of ...' warnings to identify the root cause.","Run `nodetool scrub system.batches` to clean corrupt rows and stop the warnings from recurring.","Verify disk/filesystem health (dmesg, smartctl) if IOExceptions stem from underlying hardware.","If batches were lost, have client applications re-send critical mutations or rely on their own retry/idempotency mechanisms."],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"// Operator runbook for recurring occurrences:\n// 1) grep log for 'Skipped batch replay of' to identify batch ids\n// 2) nodetool scrub system.batches\n// 3) restart replay window / verify via metrics batchlog replays","preventionTips":["Treat this as an aggregate health signal: investigate the underlying per-batch IOException cause.","Scrub system.batches after corruption-inducing events (disk full, OOM, power loss).","Ensure application writers use retries/idempotency for critical mutations.","Monitor batchlog pending tasks and replay warnings as alerting signals."],"tags":["batchlog","replay","io","durability"],"backgroundTag":"batch-replay-failed","analyzedSha":"88fd0f6a0eaed8943f05ac9e8f947882b8ddc8f1","analyzedAt":"2026-09-10T07:29:22.284Z","contentChangedAt":"2026-09-10T07:29:22.284Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}