{"record":{"id":"6c2bd0ca11ee6d30","repo":"nautechsystems/nautilus_trader","slug":"execution-payload-changed-while-clearing-rollba","errorCode":null,"errorMessage":"Execution payload {} changed while clearing rollback envelope","messagePattern":"Execution payload (.+?) changed while clearing rollback envelope","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":5730,"sourceCode":"            .await\n            .context(\"failed to recreate plaintext execution payload\")?;\n            anyhow::ensure!(\n                result.rows_affected() == 1,\n                \"Execution payload {} changed during rollback\",\n                hash.id\n            );\n            authenticate_retained_payload(&raw_transaction, &intent, &hash, keys.deployment_id())?;\n            let result = sqlx::query(\n                \"UPDATE execution_transaction_hash SET sealed_transaction = NULL, updated_at = NOW() \\\n                 WHERE id = $1 AND raw_transaction = $2 AND sealed_transaction = $3\",\n            )\n            .bind(hash.id)\n            .bind(&raw_transaction)\n            .bind(envelope)\n            .execute(&mut *transaction)\n            .await\n            .context(\"failed to clear rolled-back execution payload envelope\")?;\n            anyhow::ensure!(\n                result.rows_affected() == 1,\n                \"Execution payload {} changed while clearing rollback envelope\",\n                hash.id\n            );\n            sqlx::query(\n                \"UPDATE execution_payload_state SET progress_id = $1, updated_at = NOW() \\\n                 WHERE component = 'signed_transactions'\",\n            )\n            .bind(hash.id)\n            .execute(&mut *transaction)\n            .await\n            .context(\"failed to record execution payload rollback progress\")?;\n        }\n        transaction\n            .commit()\n            .await\n            .context(\"failed to commit execution payload rollback batch\")?;\n        Ok(false)","sourceCodeStart":5712,"sourceCodeEnd":5748,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L5712-L5748","documentation":"This is the second guard of the per-row rollback pair: after recreating the plaintext raw_transaction, the code clears sealed_transaction with an UPDATE guarded on (id, raw_transaction = unsealed value, sealed_transaction = envelope). If rows_affected != 1, the row changed between the two updates (raw value or envelope no longer match), so the batch aborts to prevent leaving the row in a double-representation or payload-less state.","triggerScenarios":"A concurrent writer modified raw_transaction or sealed_transaction for the row between the recreate-plaintext UPDATE and the clear-envelope UPDATE, or an external process cleared the envelope first. The strict guarded UPDATE then affects 0 rows and the ensure fires.","commonSituations":"Another application instance running overlapping maintenance without the shared operation lock; manual fixes applied mid-rollback; trigger-based tooling rewriting payload columns during the rollback window.","solutions":["Stop concurrent writers to execution_transaction_hash for the duration of the rollback and rerun it.","Inspect the row (id from the message) for the actual current state and repair to a single consistent representation before retrying.","Ensure every maintenance path takes the execution payload operation lock so seal/rollback steps never interleave.","Check the execution_transaction_payload_fence trigger is intact during rollback; dropping it early lets unsafe concurrent writes through."],"exampleFix":"-- diagnose\nSELECT id, raw_transaction IS NOT NULL AS has_raw, sealed_transaction IS NOT NULL AS has_sealed\nFROM execution_transaction_hash WHERE id = $offending_id;\n\n// before: interleaved writers corrupt the two-step rollback\n// writer: UPDATE execution_transaction_hash SET raw_transaction=$other WHERE id=$id;\n// rollback clear-envelope UPDATE matches 0 rows -> error\n\n// after: serialize maintenance with the operation lock before retrying\n// SELECT pg_advisory_lock(hashtext('execution_payload_maintenance'));\n// db.rollback_execution_payload(&keys, 1000).await?;\n// SELECT pg_advisory_unlock(hashtext('execution_payload_maintenance'));","handlingStrategy":"retry","validationCode":"let mid: Vec<(i64,)> = sqlx::query_as(\n    \"SELECT id FROM execution_transaction_hash \\\n     WHERE payload_expected AND sealed_transaction IS NOT NULL AND raw_transaction IS NOT NULL\",\n).fetch_all(&mut conn).await?;\nif !mid.is_empty() {\n    return Err(anyhow!(\"rollback interrupted mid-pair on rows {:?}; resolve first\", mid));\n}","typeGuard":null,"tryCatchPattern":"match db.rollback_execution_payload(&keys, batch).await {\n    Err(e) if e.to_string().contains(\"changed while clearing rollback envelope\") => {\n        // acquire exclusive maintenance lock, verify row state, retry rollback\n    },\n    other => other?,\n}","preventionTips":["Serialize all seal/rollback maintenance through a single lock holder.","Disable cron-style maintenance jobs during a manual rollback.","Re-run rollback after any interruption; it is resumable by design once writers are stopped.","Alert on any external UPDATE to execution_transaction_hash payload columns."],"tags":["database","concurrency","rollback","optimistic-concurrency","postgresql"],"backgroundTag":"internal-invariant-violation","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}