{"record":{"id":"d2f0d5d28efbb6bb","repo":"apache/seatunnel","slug":"waldisruptor-close-timeout-error","errorCode":null,"errorMessage":"WALDisruptor close timeout error","messagePattern":"WALDisruptor close timeout error","errorType":"exception","errorClass":"IMapStorageException","httpStatus":null,"severity":"error","filePath":"seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/disruptor/WALDisruptor.java","lineNumber":108,"sourceCode":"\n    public boolean tryAppendPublish(IMapFileData message, long requestId) {\n        return this.tryPublish(message, WALEventType.APPEND, requestId);\n    }\n\n    public boolean isClosed() {\n        return isClosed;\n    }\n\n    @Override\n    public void close() throws IOException {\n        // we can wait for 5 seconds, so that backlog can be committed\n        try {\n            tryPublish(null, WALEventType.CLOSED, 0L);\n            isClosed = true;\n            disruptor.shutdown(DEFAULT_CLOSE_WAIT_TIME_SECONDS, TimeUnit.SECONDS);\n        } catch (TimeoutException e) {\n            log.error(\"WALDisruptor close timeout error\", e);\n            throw new IMapStorageException(\"WALDisruptor close timeout error\", e);\n        }\n    }\n}\n","sourceCodeStart":90,"sourceCodeEnd":112,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/disruptor/WALDisruptor.java#L90-L112","documentation":"WALDisruptor.close() publishes a CLOSED event and waits up to DEFAULT_CLOSE_WAIT_TIME_SECONDS for the disruptor to drain; if shutdown exceeds that timeout this IMapStorageException is thrown. The WAL writer's background event handler did not finish pending writes within the wait window, so buffered records may not be flushed.","triggerScenarios":"Calling IMapFileStorage.close()/destroy() while the WAL event handler is blocked on slow HDFS writes (NameNode latency, network stall, datanode failure) so the disruptor ring cannot drain before the timeout.","commonSituations":"HDFS slowdown or outage at shutdown time; extremely large pending WAL buffer accumulated during a burst; thread starvation of the disruptor consumer (livelock/backpressure); long GC pauses on the storage thread.","solutions":["Check HDFS health/latency at the time of shutdown; resolve the underlying write slowness.","Retry the close operation after the storage stabilizes — buffered events may then flush.","Reduce burst write volume or increase flush frequency so less data is pending at close.","Inspect thread dumps for a blocked WALEvent handler (slow fs write, lock contention) and fix the blocking cause.","If data integrity matters, verify WAL contents after this error before deleting files in destroy()."],"exampleFix":"// before\ntry {\n    storage.close();\n} catch (IMapStorageException e) {\n    // ignored -> possible data loss\n}\n// after\ntry {\n    storage.close();\n} catch (IMapStorageException e) {\n    log.warn(\"WAL close timed out, retrying\", e);\n    Thread.sleep(5000);\n    storage.close(); // retry after HDFS recovers\n}","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try { storage.close(); } catch (IMapStorageException e) { if (e.getMessage().contains(\"close timeout\")) { /* wait for HDFS to recover, retry close; verify WAL before deleting */ } }","preventionTips":["Monitor HDFS write latency; avoid shutdowns during storage incidents","Keep WAL buffers small (frequent flush) to shorten drain time","Capture a thread dump on timeout to find the blocked handler","Never force-delete WAL files after a close timeout without verifying contents"],"tags":["timeout","wal","shutdown","hdfs"],"backgroundTag":"request-timeout","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}