{"record":{"id":"cee751e439b4eaa4","repo":"apache/seatunnel","slug":"recovery-after-rollback-prepared-transaction-failu","errorCode":null,"errorMessage":"recovery after rollback prepared transaction failure also failed, xid={}","messagePattern":"recovery after rollback prepared transaction failure also failed, xid=(.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcExactlyOnceSinkWriter.java","lineNumber":304,"sourceCode":"                            String.format(\n                                    \"failed to rollback prepared transaction after begin next transaction failure, xid=%s\",\n                                    xid),\n                            rollbackException);\n            rollbackFailure.addSuppressed(beginTxException);\n            tryRecoverPreparedTransactionsAfterRollbackFailure(xid, rollbackFailure);\n            throw rollbackFailure;\n        }\n    }\n\n    private void tryRecoverPreparedTransactionsAfterRollbackFailure(\n            Xid failedRollbackXid, JdbcConnectorException rollbackFailure) {\n        try {\n            LOG.warn(\n                    \"rollback prepared transaction failed, try to recover pending transactions for current subtask, xid={}\",\n                    failedRollbackXid);\n            xaGroupOps.recoverAndRollback(context, sinkcontext, xidGenerator, null);\n        } catch (Exception recoveryException) {\n            LOG.warn(\n                    \"recovery after rollback prepared transaction failure also failed, xid={}\",\n                    failedRollbackXid,\n                    recoveryException);\n            rollbackFailure.addSuppressed(recoveryException);\n        }\n    }\n\n    private void failAndRollbackCurrentXidQuietly() {\n        if (currentXid == null || !xaFacade.isOpen()) {\n            return;\n        }\n        Xid xid = currentXid;\n        try {\n            LOG.debug(\"remove current transaction, xid={}\", xid);\n            xaFacade.failAndRollback(xid);\n        } catch (Exception e) {\n            LOG.warn(\"unable to fail/rollback current transaction, xid={}\", xid, e);\n        } finally {","sourceCodeStart":286,"sourceCodeEnd":322,"githubUrl":"https://github.com/apache/seatunnel/blob/cf67b549a7a6c35fa0beb12d83c62892427ea919/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/sink/JdbcExactlyOnceSinkWriter.java#L286-L322","documentation":"Logged by tryRecoverPreparedTransactionsAfterRollbackFailure when the fallback XA recovery itself throws: both the direct rollback of the prepared transaction and the recoverAndRollback sweep have failed. The recovery exception is attached as suppressed to the pending JdbcConnectorException, which is then thrown, so the transaction likely remains in-doubt on the database until manual or later recovery.","triggerScenarios":"rollbackPrepareXidOrThrow -> tryRecoverPreparedTransactionsAfterRollbackFailure where xaGroupOps.recoverAndRollback(context, sinkcontext, xidGenerator, null) throws — typically because the connection/facade is still broken, the DB rejects XA RECOVER (missing privileges), or the resource manager is unreachable.","commonSituations":"Prolonged DB outage during checkpoint; DB account lacking recovery privileges (MySQL XA_RECOVER_ADMIN); XA connection still bound to the dead session; driver/dialect not fully supporting XA recovery listing.","solutions":["Read the full thrown JdbcConnectorException: beginTxException + rollbackFailure + suppressed recoveryException pinpoint the root cause (usually connectivity)","Restore DB connectivity, then restart the job — startup recovery will sweep and resolve in-doubt prepared transactions","Grant recovery privileges: MySQL 8 requires XA_RECOVER_ADMIN for XA RECOVER; verify with a manual 'xa recover'","Manually resolve in-doubt transactions with database tooling (xa rollback '<xid>' in MySQL, COMMIT FORCE/ROLLBACK FORCE in Oracle) to avoid locks and orphaned transactions","Check dialect XA support and driver version; some drivers mis-report recovery lists — upgrade to a driver with solid XA support"],"exampleFix":"// before: user without recovery privilege\nCREATE USER 'st'@'%' IDENTIFIED BY '***';\nGRANT SELECT, INSERT, UPDATE ON db.* TO 'st'@'%';\n// after\nCREATE USER 'st'@'%' IDENTIFIED BY '***';\nGRANT SELECT, INSERT, UPDATE ON db.* TO 'st'@'%';\nGRANT XA_RECOVER_ADMIN ON *.* TO 'st'@'%';","handlingStrategy":"try-catch","validationCode":"// verify recovery listing works with the configured account BEFORE running production jobs\n-- MySQL: xa recover;  (needs XA_RECOVER_ADMIN)\n-- Oracle: select * from dba_pending_transactions;\n-- PostgreSQL: select gid from pg_prepared_xacts;","typeGuard":null,"tryCatchPattern":"try {\n    writer.prepareCommit();\n} catch (JdbcConnectorException e) {\n    // contains beginTxException + rollbackFailure + suppressed recoveryException\n    for (Throwable t : e.getSuppressed()) {\n        LOG.error(\"recovery also failed, resolve in-doubt xids manually\", t);\n    }\n    // restore connectivity, then restart so startup recovery retries\n}","preventionTips":["Grant XA_RECOVER_ADMIN (MySQL 8) or equivalent recovery privileges to the connector user","Alert on pg_prepared_xacts / xa recover listing growth — orphaned in-doubt transactions hold locks","Keep DB connectivity stable (keepalive, sane timeouts) so both rollback and recovery paths stay reachable","Use a driver/dialect with well-tested XA recovery support; upgrade old JDBC drivers","Document a manual runbook for resolving in-doubt transactions after prolonged outages"],"tags":["jdbc","xa","recovery","distributed-transaction","in-doubt-transaction"],"backgroundTag":"xa-recovery-failed","analyzedSha":"cf67b549a7a6c35fa0beb12d83c62892427ea919","analyzedAt":"2026-09-10T21:44:55.265Z","contentChangedAt":"2026-09-10T21:44:55.265Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}