{"record":{"id":"cee132ee36013b92","repo":"nautechsystems/nautilus_trader","slug":"execution-payload-changed-during-rollback","errorCode":null,"errorMessage":"Execution payload {} changed during rollback","messagePattern":"Execution payload (.+?) changed during rollback","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":5714,"sourceCode":"            let envelope = hash\n                .sealed_transaction\n                .as_deref()\n                .expect(\"rollback query requires envelope\");\n            let intent = load_execution_intent(&mut transaction, hash.intent_id).await?;\n            let context = payload_context(&intent, &hash, keys.deployment_id())?;\n            let raw_transaction = keys.unseal(envelope, &context)?;\n            authenticate_retained_payload(&raw_transaction, &intent, &hash, keys.deployment_id())?;\n            let result = sqlx::query(\n                \"UPDATE execution_transaction_hash SET raw_transaction = $2, updated_at = NOW() \\\n                 WHERE id = $1 AND raw_transaction IS NULL AND sealed_transaction = $3\",\n            )\n            .bind(hash.id)\n            .bind(&raw_transaction)\n            .bind(envelope)\n            .execute(&mut *transaction)\n            .await\n            .context(\"failed to recreate plaintext execution payload\")?;\n            anyhow::ensure!(\n                result.rows_affected() == 1,\n                \"Execution payload {} changed during rollback\",\n                hash.id\n            );\n            authenticate_retained_payload(&raw_transaction, &intent, &hash, keys.deployment_id())?;\n            let result = sqlx::query(\n                \"UPDATE execution_transaction_hash SET sealed_transaction = NULL, updated_at = NOW() \\\n                 WHERE id = $1 AND raw_transaction = $2 AND sealed_transaction = $3\",\n            )\n            .bind(hash.id)\n            .bind(&raw_transaction)\n            .bind(envelope)\n            .execute(&mut *transaction)\n            .await\n            .context(\"failed to clear rolled-back execution payload envelope\")?;\n            anyhow::ensure!(\n                result.rows_affected() == 1,\n                \"Execution payload {} changed while clearing rollback envelope\",","sourceCodeStart":5696,"sourceCodeEnd":5732,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L5696-L5732","documentation":"After unsealing the envelope and writing the plaintext raw_transaction, the code verifies exactly one row was affected by an UPDATE guarded on (id, raw_transaction IS NULL, sealed_transaction = envelope). Zero affected rows means the row changed underneath the rollback — the expected snapshot no longer matches — so the batch aborts to avoid overwriting a concurrent modification with a stale unsealed payload.","triggerScenarios":"Between the SELECT ... FOR UPDATE batch read and the UPDATE, the row's sealed_transaction or raw_transaction was modified by another connection (possible if lock ordering with the fence trigger/advisory lock was bypassed), or a manual edit cleared/moved the payload. Rows_affected == 0 on the guarded recreate-plaintext UPDATE throws this error.","commonSituations":"Another node or an operator running conflicting maintenance without the payload operation lock; manual SQL updates to execution_transaction_hash during rollback; an application bug writing transaction payloads concurrently with a rollback window.","solutions":["Stop all other writers/maintenance on execution_transaction_hash while the rollback runs.","Verify the row (id from the message): confirm its current sealed_transaction/raw_transaction values and whether another process altered them.","Rerun the rollback from the current state once concurrent writers are stopped; the FOR UPDATE re-read will pick up the new snapshot.","Ensure all maintenance paths acquire the execution payload operation lock (lock_execution_payload_operation) before touching these rows."],"exampleFix":"// before: concurrent write racing the rollback UPDATE\n// some_tool: UPDATE execution_transaction_hash SET sealed_transaction=$new WHERE id=$id; -- no lock\n// rollback: rows_affected == 0 -> \"Execution payload {id} changed during rollback\"\n\n// after: hold the maintenance lock for the whole window\n// SELECT pg_advisory_lock(hashtext('execution_payload_maintenance'));\n// ... run rollback_execution_payload ... \n// SELECT pg_advisory_unlock(hashtext('execution_payload_maintenance'));","handlingStrategy":"retry","validationCode":"let locked: (String,) = sqlx::query_as(\n    \"SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions' FOR UPDATE\",\n).fetch_optional(&mut conn).await?.ok_or_else(|| anyhow!(\"payload protection not active\"))?;\n// proceed only while holding the maintenance advisory lock and state is 'rollback'","typeGuard":null,"tryCatchPattern":"match db.rollback_execution_payload(&keys, batch).await {\n    Err(e) if e.to_string().contains(\"changed during rollback\") => {\n        // stop concurrent writers, then retry the rollback once\n        db.rollback_execution_payload(&keys, batch).await?;\n    },\n    other => other?,\n}","preventionTips":["Hold a database advisory lock for the entire rollback window.","Freeze all payload-writing jobs and manual maintenance during rollback.","Run rollback on only one node/instance at a time.","Verify the fence trigger exists before starting."],"tags":["database","concurrency","rollback","optimistic-concurrency","postgresql"],"backgroundTag":"internal-invariant-violation","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}