{"record":{"id":"a713f591a4331026","repo":"nautechsystems/nautilus_trader","slug":"execution-payload-storage-is-not-rolling-back","errorCode":null,"errorMessage":"Execution payload storage is not rolling back","messagePattern":"Execution payload storage is not rolling back","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":5633,"sourceCode":"        batch_size: i64,\n    ) -> anyhow::Result<bool> {\n        let mut transaction = self\n            .pool\n            .begin()\n            .await\n            .context(\"failed to start execution payload rollback batch\")?;\n        lock_execution_payload_operation(&mut transaction).await?;\n        let state_row = sqlx::query(\n            \"SELECT deployment_id, protocol_version, operation, active_key_id \\\n             FROM execution_payload_state WHERE component = 'signed_transactions' FOR UPDATE\",\n        )\n        .fetch_optional(&mut *transaction)\n        .await\n        .context(\"failed to lock execution payload rollback state\")?\n        .ok_or_else(|| anyhow::anyhow!(\"Execution payload rollback state is missing\"))?;\n        let state = execution_payload_state_from_row(&state_row)?;\n        validate_execution_payload_state(&state, keys)?;\n        anyhow::ensure!(\n            state.operation == \"rollback\",\n            \"Execution payload storage is not rolling back\"\n        );\n        let rows = sqlx::query_as::<_, ExecutionTransactionHashRow>(\n            \"\n            SELECT\n                id, intent_id, chain_id, transaction_hash, payload_expected,\n                raw_transaction, sealed_transaction, status, block_number, block_hash,\n                receipt_success, gas_used, effective_gas_price, current\n            FROM execution_transaction_hash\n            WHERE payload_expected AND sealed_transaction IS NOT NULL\n            ORDER BY id\n            LIMIT $1\n            FOR UPDATE\n            \",\n        )\n        .bind(batch_size)\n        .fetch_all(&mut *transaction)","sourceCodeStart":5615,"sourceCodeEnd":5651,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L5615-L5651","documentation":"This error comes from the batched execution-payload rollback in the blockchain cache database. Each batch transaction re-reads the execution_payload_state row (locked FOR UPDATE) and asserts its operation column equals 'rollback'. If the state row shows anything other than 'rollback', the storage layer has concurrently moved out of rollback mode (e.g. back to 'ready' or into another maintenance operation), and continuing would corrupt the two-representation (raw/sealed) payload migration.","triggerScenarios":"Calling rollback_execution_payload (which begins a rollback then loops rollback_execution_payload_batch) while another process concurrently completes, re-initializes, or re-enters the payload protection workflow, changing execution_payload_state.operation from 'rollback' to something else between the begin and batch phases. Also occurs if the state row was manually edited or a maintenance/seal operation was started mid-rollback.","commonSituations":"Two operators or two application instances running payload seal/rollback maintenance at the same time against the same database; a manual SQL fix-up of execution_payload_state during an in-flight rollback; resuming a rollback after another host already finished it and reset state; stale orchestration scripts racing the app's own migration job.","solutions":["Check the execution_payload_state table (component='signed_transactions') to see the current operation value; ensure only one maintenance workflow is running.","Take an advisory/application-level lock so seal and rollback maintenance cannot run concurrently, then retry the rollback.","If the rollback actually completed elsewhere, do not resume; verify row consistency and let the normal path proceed.","If the state was manually modified, restore operation='rollback' (or restart the rollback from a consistent 'ready' state) before retrying."],"exampleFix":"// before: two services racing\n// service A: db.rollback_execution_payload(&keys, 1000).await?;\n// service B (concurrently): db.seal_execution_payload(...).await?;\n\n// after: serialize maintenance with an advisory lock\nlet mut conn = db.pool.acquire().await?;\nsqlx::query(\"SELECT pg_advisory_lock(hashtext('execution_payload_maintenance'))\")\n    .execute(&mut *conn).await?;\ndb.rollback_execution_payload(&keys, 1000).await?;\nsqlx::query(\"SELECT pg_advisory_unlock(hashtext('execution_payload_maintenance'))\")\n    .execute(&mut *conn).await?;","handlingStrategy":"validation","validationCode":"let state: (String,) = sqlx::query_as(\n    \"SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions'\",\n).fetch_one(&mut conn).await?;\nif state.0 != \"rollback\" && state.0 != \"ready\" {\n    return Err(anyhow!(\"payload storage in '{}' maintenance; rollback not safe\", state.0));\n}","typeGuard":null,"tryCatchPattern":"match db.rollback_execution_payload(&keys, batch).await {\n    Ok(()) => {},\n    Err(e) if e.to_string().contains(\"not rolling back\") => {\n        // another maintenance operation holds/changed the state; serialize and retry later\n    },\n    Err(e) => return Err(e),\n}","preventionTips":["Run only one payload maintenance workflow at a time, guarded by a database advisory lock.","Never hand-edit execution_payload_state while the application is running.","Monitor the operation column and alert on unexpected transitions.","Ensure all nodes run the same application version before starting seal/rollback maintenance."],"tags":["database","rollback","concurrency","state-machine","postgresql"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}