{"record":{"id":"24614e0915f4f7a6","repo":"apache/seatunnel","slug":"wal-send-failed-for-record-id-batchid-row","errorCode":null,"errorMessage":"WAL send failed for record id={}, batchId={}; row stays SENDING for resurrect/retry","messagePattern":"WAL send failed for record id=(.+?), batchId=(.+?); row stays SENDING for resurrect/retry","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"seatunnel-edge-agent/seatunnel-edge-agent-starter/src/main/java/org/apache/seatunnel/edge/agent/starter/runtime/EdgeAgentRuntimeScheduler.java","lineNumber":190,"sourceCode":"        int sent = sendClaimedRecords();\n        return appended > 0 || sent > 0 || !events.isEmpty();\n    }\n\n    private int sendClaimedRecords() throws Exception {\n        List<WalRecord> records = walStore.claimPending(maxPollRecords, maxAttempts);\n        int sent = 0;\n        for (WalRecord record : records) {\n            long batchId = record.getBatchId() > 0 ? record.getBatchId() : record.getId();\n            try {\n                transport.send(batchId, payloadSerializer.serialize(record.getPayload()));\n                walStore.ack(record.getId());\n                sent++;\n                sendFailureAttempt = 0;\n            } catch (IOException ex) {\n                if (isDecryptFailed(ex)) {\n                    throw ex;\n                }\n                LOG.warn(\n                        \"WAL send failed for record id={}, batchId={}; row stays SENDING for\"\n                                + \" resurrect/retry\",\n                        record.getId(),\n                        batchId,\n                        ex);\n                Thread.sleep(\n                        EdgeTransportConfig.computeBackoffMillis(\n                                sendFailureAttempt++, backoffMs, backoffMaxMs));\n                break;\n            } catch (InterruptedException ex) {\n                Thread.currentThread().interrupt();\n                throw ex;\n            }\n        }\n        return sent;\n    }\n\n    private static boolean isDecryptFailed(IOException ex) {","sourceCodeStart":172,"sourceCodeEnd":208,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-edge-agent/seatunnel-edge-agent-starter/src/main/java/org/apache/seatunnel/edge/agent/starter/runtime/EdgeAgentRuntimeScheduler.java#L172-L208","documentation":"EdgeAgentRuntimeScheduler.sendClaimedRecords catches IOException while sending a WAL record to the remote endpoint. Non-decrypt failures are logged as a warning: the record deliberately stays in SENDING state so a later resurrection/retry pass can resend it. Decrypt failures are rethrown because they indicate a non-transient problem.","triggerScenarios":"sendClaimedRecords -> sending a claimed WAL record over the transport throws IOException (connection drop, socket reset, broken pipe) that is not classified as a decrypt failure.","commonSituations":"Receiver restarted mid-batch, network partition between agent and receiver, receiver overloaded closing sockets, or TLS issues corrupting the stream.","solutions":["Check the logged record id/batchId and exception to confirm it is a transient network issue.","Verify the receiver endpoint is reachable and healthy (ports, auth, load).","Wait for the retry/resurrect mechanism to resend the SENDING record; ensure backoff settings are sane.","If failures persist, restart the scheduler after confirming connectivity; check that decrypt keys/config are correct if the error ever surfaces as decrypt-failed."],"exampleFix":"// before: no retry tuning, short timeout kills long sends\ntransport {\n  timeout-ms = 1000\n}\n// after: tolerate slower links\ntransport {\n  timeout-ms = 30000\n  max-backoff-ms = 60000\n}","handlingStrategy":"retry","validationCode":"// Pre-send health probe\nif (!transport.isSessionHealthy()) {\n    transport.ensureAuthenticatedSession();\n}","typeGuard":null,"tryCatchPattern":"try {\n    send(record);\n} catch (IOException ex) {\n    if (!isDecryptFailed(ex)) {\n        // leave record in SENDING; rely on resurrect/retry with capped backoff\n        backoffSleep(attempt++);\n    } else {\n        throw ex; // non-transient: surface immediately\n    }\n}","preventionTips":["Keep SENDING-state resurrect/retry enabled and monitor retry depth.","Alert on receiver health so sockets are not dropped mid-batch.","Set realistic transport timeouts for batch sizes on your network.","Separate decrypt/config errors (fail fast) from transport errors (retry)."],"tags":["network","io","wal","retry"],"backgroundTag":"network-request-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}