apache/seatunnel · warning
Scheduler thread did not terminate after shutdownNow(); skip
Error message
Scheduler thread did not terminate after shutdownNow(); skipping local buffer drain to avoid concurrent writer access.
What it means
If the scheduler thread is still alive even after shutdownNow(), close() skips draining the local buffer because doing so would race with the scheduler on writerLock. It logs this warning and instead safely closes the current trace file writer under the lock.
Source
Thrown at seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/event/JobEventLocalFileHandler.java:440
log.warn(
"Scheduler timed out during close; up to {} ringbuffer event(s) were"
+ " not flushed to disk. Local buffer (cap={}) will be drained"
+ " as fallback.",
unsynced,
LOCAL_EVENT_BUFFER_CAPACITY);
}
} catch (Exception e) {
log.warn(
"Scheduler timed out during close; relying on local buffer as fallback."
+ " Could not determine dropped event count: {}",
e.getMessage());
}
if (!finallyTerminated) {
// The scheduler thread is still alive even after shutdownNow(). Draining
// the local buffer here would race with the scheduler on writerLock.
// closing=true already prevents the scheduler from opening new files,
// so we can safely close the current writer under the lock and return.
log.warn(
"Scheduler thread did not terminate after shutdownNow();"
+ " skipping local buffer drain to avoid concurrent writer access.");
synchronized (writerLock) {
TraceFileWriter writer = currentWriter;
currentWriter = null;
if (writer != null) {
try {
writer.close();
} catch (IOException e) {
log.error("Failed to close current writer", e);
}
}
}
return;
}
try {
reportFromRingbuffer(true);
} catch (HazelcastInstanceNotActiveException e) {View on GitHub (pinned to cf67b549a7)
Solutions
- Diagnose why the scheduler thread is blocked — take a thread dump (jstack) at shutdown to find the stuck frame.
- Check for filesystem hangs (NFS timeouts, disk full, permission issues) on the trace output directory.
- Pending local-buffer events are dropped in this path; reduce event volume or fix I/O to avoid hitting it.
Example fix
null
Defensive patterns
Strategy: try-catch
Try / catch
try { handler.close(); } catch (Exception e) { log.warn("scheduler thread stuck; local buffer drain skipped", e); } Prevention
- Capture thread dumps on slow shutdowns to find blocked scheduler threads.
- Avoid network filesystems for trace output.
- Verify disk space and permissions on the trace directory.
When it happens
Trigger: JobEventLocalFileHandler.close() awaited scheduler termination, timed out, then called shutdownNow(), and finallyTerminated is still false — a stuck or blocked scheduler thread (e.g. blocked on file I/O or an internal lock).
Common situations: Scheduler thread blocked on a hung file system (NFS, full disk) or a deadlock during cluster shutdown; events still in the local buffer are intentionally not written to avoid corruption.
Understand the failure class
Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.
Related errors
- Scheduler timed out during close; up to {} ringbuffer event(
- Continuous discovery scheduler does not terminate in 5 secon
- Timed out waiting for http report scheduler to stop
- Scheduler timed out during close; relying on local buffer as
- Unsafe invoke, the current thread[%s] has not acquired the l
AI-assisted analysis of apache/seatunnel@cf67b549a7 (2026-09-10).
Data as JSON: /api/errors/0012106285e48285.
Report an issue: GitHub.