Out of journal flushes over the past s with average…

Error message

Out of {} {} journal flushes over the past {}s with average duration of {}ms, {} have exceeded the configured flush period by an average of {}ms

What it means

Journal's Flusher tracks fsync durations; after each fsync (afterFSync -> processDuration) it detects flushes that overran the configured flush period. When enough flushes lag, it emits this rate-limited (noSpamLogger) warning summarizing how many of the recent flushes exceeded the period and by how much on average. It signals the journal cannot keep up with its flush interval, typically due to slow disk.

Solutions

Check disk latency (iostat -x) for the journal volume; move journal/commitlog to dedicated fast storage.
Reduce concurrent I/O (throttle compaction throughput) competing with fsyncs.
Increase the journal flush period so it matches achievable fsync latency.
Verify the filesystem supports fast fsync (avoid overlays/network mounts).
If the warning is sparse and average lag is small, treat as informational and monitor.

Example fix

// before (cassandra.yaml)
# flush_period: 1000ms  # fsyncs take ~50ms, unrealistic
// after
# flush_period: 10000ms  # headroom above measured avg fsync duration

Defensive patterns

Strategy: validation

Validate before calling

// before tuning: measure achievable fsync latency on the journal volume
long start = System.nanoTime();
try (java.io.FileChannel ch = java.io.FileChannel.open(journalDir.resolve("f.dat"), java.nio.file.StandardOpenOption.WRITE)) {
    ch.force(true);
}
long fsyncMs = (System.nanoTime() - start) / 1_000_000;
if (fsyncMs > configuredFlushPeriodMs) reconfigureFlushPeriod(fsyncMs * 2);

Prevention

Monitor disk utilization and fsync latency (iostat -x, nodetool tpstats) continuously.
Isolate journal/commitlog on a dedicated low-latency volume.
Set flush period with headroom above measured fsync latency.
Throttle compaction and streaming to reduce I/O contention.

When it happens

Trigger: Periodic journal flusher runs where fsync latency pushes average flush duration past the configured flush period; sustained fsync counts with average duration exceeding the target interval trigger the warning once per no-spam window.

Common situations: Disk saturation from compaction or other I/O on the same volume; journal and data sharing a slow EBS/network disk; fsync latency spikes; flush period tuned too aggressively for the storage.

Related errors

AI-assisted analysis of apache/cassandra@88fd0f6a0e (2026-09-10). Data as JSON: /api/errors/25977b785e9fee90. Report an issue: GitHub.

Appendix: source

Thrown at src/java/org/apache/cassandra/journal/Flusher.java:382

        private void processDuration(long startedFlushAt, long finishedFsyncAt)
        {
            fsyncCount++;
            duration += (finishedFsyncAt - startedFlushAt);

            long flushPeriodNanos = flushPeriodNanos();
            long lag = finishedFsyncAt - (startedFlushAt + flushPeriodNanos);
            if (flushPeriodNanos <= 0 || lag <= 0)
                return;

            lagCount++;
            lagDuration += lag;

            if (firstLaggedAt == Long.MIN_VALUE)
                firstLaggedAt = finishedFsyncAt;

            boolean logged =
            noSpamLogger.warn(finishedFsyncAt,
                              "Out of {} {} journal flushes over the past {}s with average duration of {}ms, " +
                              "{} have exceeded the configured flush period by an average of {}ms",
                              fsyncCount,
                              journal.name,
                              format("%.2f", (finishedFsyncAt - firstLaggedAt) * 1e-9d),
                              format("%.2f", duration * 1e-6d / fsyncCount),
                              lagCount,
                              format("%.2f", lagDuration * 1e-6d / lagCount));

            if (logged) // reset metrics for next log statement
            {
                firstLaggedAt = Long.MIN_VALUE;
                fsyncCount = lagCount = 0;
                duration = lagDuration = 0;
            }
        }

        private void afterFSync(long startedAt, long segment, int position)

View on GitHub (pinned to 88fd0f6a0e)