{"record":{"id":"c4447721afe3e6c6","repo":"apache/seatunnel","slug":"error-sink-writer-failed-during-shutdown","errorCode":null,"errorMessage":"Error sink writer failed during shutdown","messagePattern":"Error sink writer failed during shutdown","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/error/DefaultErrorSinkWriter.java","lineNumber":533,"sourceCode":"                try {\n                    synchronized (writerLock) {\n                        sinkWriter.write(row);\n                    }\n                } catch (Throwable writeEx) {\n                    workerFailure = writeEx;\n                    throw writeEx;\n                } finally {\n                    pendingRows.decrementAndGet();\n                }\n            }\n        } catch (Throwable e) {\n            if (e instanceof Error) {\n                workerFailure = e;\n                throw (Error) e;\n            }\n            workerFailure = e;\n            if (closed) {\n                log.warn(\"Error sink writer failed during shutdown\", e);\n            } else {\n                log.error(\"Error sink writer failed\", e);\n            }\n        } finally {\n            try {\n                if (sinkWriter != null) {\n                    sinkWriter.close();\n                }\n            } catch (Throwable closeEx) {\n                if (closeEx instanceof Error) {\n                    throw (Error) closeEx;\n                }\n                if (workerFailure != null) {\n                    workerFailure.addSuppressed(closeEx);\n                } else {\n                    workerFailure = closeEx;\n                }\n                log.warn(\"Failed to close error sink writer\", closeEx);","sourceCodeStart":515,"sourceCodeEnd":551,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/task/error/DefaultErrorSinkWriter.java#L515-L551","documentation":"The DefaultErrorSinkWriter drain loop caught a Throwable from sink writing or lifecycle. Because the writer is already in the closed state, the failure happened during shutdown and is logged at WARN instead of ERROR. The Throwable is recorded as workerFailure and rethrown if it is an Error; data still in the queue at this point may not be written.","triggerScenarios":"drainLoop processes records or closes the underlying sinkWriter after close() has set closed=true, and an exception occurs (e.g. sink write fails, sinkWriter.close() fails, I/O error while draining remaining records).","commonSituations":"Job cancellation/stop races with an in-flight error-sink write; error sink connection dropped right at shutdown; sink close() throwing; disk/network failure during final drain.","solutions":["Inspect the logged underlying exception 'e' for the real sink-side cause and fix that (connectivity, schema, auth).","Retry the job or re-send the error records that were pending at shutdown, since they may be lost.","Gracefully cancel jobs so the drain loop empties the queue before close.","Harden the error sink connector (timeouts, retries) so shutdown writes succeed."],"exampleFix":"// before: hard-cancel drops drain during shutdown\nseaTunnel.sh -can <jobId>  // immediate cancel while sink blocked\n// after: stop gracefully / with timeout so drain completes\nseaTunnel.sh -s <jobId>   // graceful stop, drain loop finishes first","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"try {\n    errorSinkWriter.write(record);\n} catch (Exception e) {\n    log.error(\"error record dropped\", e);\n    deadLetterQueue.add(record); // preserve data\n}","preventionTips":["Always stop jobs gracefully so the drain loop can finish before close.","Persist the queue externally (e.g. Kafka error topic) if loss during shutdown is unacceptable.","Monitor this warning; each occurrence means un-drained error rows may be dropped.","Harden the underlying sink connector with retries/timeouts."],"tags":["error-sink","shutdown-race","data-loss","drain-loop"],"backgroundTag":"shutdown-race-condition","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}