Encountered unexpected exceptions while sending out batches

Error message

Encountered %d unexpected exceptions while sending out batches

What it means

At the end of processBatchlogEntries, if any batches were skipped during the replay pass (each logged individually), BatchlogManager logs this aggregated warning with the skip count plus the retained exception. It signals that batchlog guarantees could not be fully honored for `skipped` batches; hints for other batches are still flushed and fsynced before those batches are deleted.

Solutions

Read the attached cause (caughtException) and the per-batch 'Skipped batch replay of ...' warnings to identify the root cause.
Run `nodetool scrub system.batches` to clean corrupt rows and stop the warnings from recurring.
Verify disk/filesystem health (dmesg, smartctl) if IOExceptions stem from underlying hardware.
If batches were lost, have client applications re-send critical mutations or rely on their own retry/idempotency mechanisms.

Defensive patterns

Strategy: retry

Try / catch

// Operator runbook for recurring occurrences:
// 1) grep log for 'Skipped batch replay of' to identify batch ids
// 2) nodetool scrub system.batches
// 3) restart replay window / verify via metrics batchlog replays

Prevention

Treat this as an aggregate health signal: investigate the underlying per-batch IOException cause.
Scrub system.batches after corruption-inducing events (disk full, OOM, power loss).
Ensure application writers use retries/idempotency for critical mutations.
Monitor batchlog pending tasks and replay warnings as alerting signals.

When it happens

Trigger: One or more dispatchBatch calls threw IOException during replayFailedBatches, incrementing the skipped counter — corrupt batchlog payloads, deserialization failures, or I/O errors.

Common situations: Clusters recovering from disk corruption or unclean shutdowns; replay after upgrades where stored batches were written with incompatible versions; repeated warnings across replays indicate persistent corruption in system.batches.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/02ac4af6756320dc. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/batchlog/BatchlogManager.java:337

            }

            if (++positionInPage == pageSize)
            {
                // We have reached the end of a batch. To avoid keeping more than a page of mutations in memory,
                // finish processing the page before requesting the next row.
                finishAndClearBatches(unfinishedBatches, hintedNodes, replayedBatches);
                positionInPage = 0;
            }
        }

        // finalize the incomplete last page of batches
        if (positionInPage > 0)
            finishAndClearBatches(unfinishedBatches, hintedNodes, replayedBatches);
        else
            logger.trace("Had no batches to replay");

        if (caughtException != null)
            logger.warn(String.format("Encountered %d unexpected exceptions while sending out batches", skipped), caughtException);

        // to preserve batch guarantees, we must ensure that hints (if any) have made it to disk, before deleting the batches
        HintsService.instance.flushAndFsyncBlockingly(hintedNodes);

        // once all generated hints are fsynced, actually delete the batches
        replayedBatches.forEach(BatchlogManager::remove);
    }

    private void dispatchBatch(RateLimiter rateLimiter, Row row, TimeUUID id, int version, Set<UUID> hintedNodes, ArrayList<ReplayingBatch> unfinishedBatches) throws IOException
    {
        while (true)
        {
            ClusterMetadata cm = ClusterMetadata.current();
            try
            {
                ReplayingBatch batch = new ReplayingBatch(id, version, row.getList("mutations", BytesType.instance), cm);
                if (batch.replay(rateLimiter, hintedNodes))
                {

View on GitHub (pinned to 88fd0f6a0e)