apache/seatunnel · error · IMapStorageException

WALDisruptor close timeout error

Error message

WALDisruptor close timeout error

What it means

WALDisruptor.close() publishes a CLOSED event and waits up to DEFAULT_CLOSE_WAIT_TIME_SECONDS for the disruptor to drain; if shutdown exceeds that timeout this IMapStorageException is thrown. The WAL writer's background event handler did not finish pending writes within the wait window, so buffered records may not be flushed.

Source

Thrown at seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/disruptor/WALDisruptor.java:108

    public boolean tryAppendPublish(IMapFileData message, long requestId) {
        return this.tryPublish(message, WALEventType.APPEND, requestId);
    }

    public boolean isClosed() {
        return isClosed;
    }

    @Override
    public void close() throws IOException {
        // we can wait for 5 seconds, so that backlog can be committed
        try {
            tryPublish(null, WALEventType.CLOSED, 0L);
            isClosed = true;
            disruptor.shutdown(DEFAULT_CLOSE_WAIT_TIME_SECONDS, TimeUnit.SECONDS);
        } catch (TimeoutException e) {
            log.error("WALDisruptor close timeout error", e);
            throw new IMapStorageException("WALDisruptor close timeout error", e);
        }
    }
}

View on GitHub (pinned to cf67b549a7)

Solutions

1. Check HDFS health/latency at the time of shutdown; resolve the underlying write slowness.
2. Retry the close operation after the storage stabilizes — buffered events may then flush.
3. Reduce burst write volume or increase flush frequency so less data is pending at close.
4. Inspect thread dumps for a blocked WALEvent handler (slow fs write, lock contention) and fix the blocking cause.
5. If data integrity matters, verify WAL contents after this error before deleting files in destroy().

Example fix

// before
try {
    storage.close();
} catch (IMapStorageException e) {
    // ignored -> possible data loss
}
// after
try {
    storage.close();
} catch (IMapStorageException e) {
    log.warn("WAL close timed out, retrying", e);
    Thread.sleep(5000);
    storage.close(); // retry after HDFS recovers
}

try { storage.close(); } catch (IMapStorageException e) { if (e.getMessage().contains("close timeout")) { /* wait for HDFS to recover, retry close; verify WAL before deleting */ } }

When it happens

Trigger: Calling IMapFileStorage.close()/destroy() while the WAL event handler is blocked on slow HDFS writes (NameNode latency, network stall, datanode failure) so the disruptor ring cannot drain before the timeout.

Common situations: HDFS slowdown or outage at shutdown time; extremely large pending WAL buffer accumulated during a burst; thread starvation of the disruptor consumer (livelock/backpressure); long GC pauses on the storage thread.

Understand the failure class

Background: Request timed out: what client-side request timeouts mean across libraries (Request timed out, TIMED_OUT, APITimeoutError) — this error's family across 39 libraries.

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• Failed to close the source reader in {} ms. There are still
• Edge socket receiver executor did not terminate within timeo
• Continuous discovery scheduler does not terminate in 5 secon
• Timed out waiting for http report scheduler to stop
• Scheduler timed out during close; up to {} ringbuffer event(

AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10). Data as JSON: /api/errors/d2f0d5d28efbb6bb. Report an issue: GitHub.