{"record":{"id":"4fef2359897199a9","repo":"nautechsystems/nautilus_trader","slug":"execution-payload-storage-is-in-operation-mainte","errorCode":null,"errorMessage":"Execution payload storage is in {operation} maintenance; complete or roll back that operation before connecting","messagePattern":"Execution payload storage is in (.+?) maintenance; complete or roll back that operation before connecting","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":4638,"sourceCode":"            ),\n        }\n\n        loop {\n            let state = self.execution_payload_state().await?.ok_or_else(|| {\n                anyhow::anyhow!(\"Execution payload marker exists without durable state\")\n            })?;\n            validate_execution_payload_state(&state, keys)?;\n            match state.operation.as_str() {\n                \"migrate\" => {\n                    if self\n                        .migrate_execution_payload_batch(keys, EXECUTION_PAYLOAD_BATCH_SIZE)\n                        .await?\n                    {\n                        break;\n                    }\n                }\n                \"ready\" => break,\n                operation => anyhow::bail!(\n                    \"Execution payload storage is in {operation} maintenance; complete or roll back that operation before connecting\"\n                ),\n            }\n        }\n\n        self.validate_execution_payload_ready(keys).await\n    }\n\n    /// Requires ready protected storage and authenticates every persisted signed payload.\n    pub(crate) async fn require_execution_payload_storage(\n        &self,\n        keys: &PayloadKeySet,\n        policy: PayloadPolicy,\n        batch_size: i64,\n    ) -> anyhow::Result<ExecutionPayloadLease> {\n        self.require_execution_payload_storage_ready(keys).await?;\n        let (check, transaction) = self\n            .inspect_execution_payload_storage(Some(keys), Some(policy), batch_size)","sourceCodeStart":4620,"sourceCodeEnd":4656,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L4620-L4656","documentation":"ensure_execution_payload_storage connects to protected signed-transaction storage and processes any pending maintenance operation recorded in execution_payload_state. When the stored operation is neither 'migrate' nor 'ready' (e.g. 'rewrap' or 'rollback'), it refuses to connect because the storage schema is mid-transition and must be completed or rolled back first. This is a safety gate so connections never observe half-encrypted payload rows.","triggerScenarios":"Calling ensure_execution_payload_storage (normal node startup/connect path) while execution_payload_state.operation holds a maintenance value such as 'rewrap' or 'rollback' left over from an interrupted or in-progress key rotation or protection rollback.","commonSituations":"A key-rotation (rewrap) or rollback run crashed or was killed midway and its state row never returned to 'ready'; a second node instance started while another operator was running rotation tooling; manual maintenance was started via admin scripts and forgotten.","solutions":["Resume/complete the in-progress operation by running the corresponding rotation/rollback command (rewrap_execution_payload or rollback path) until state.operation becomes 'ready'","Alternatively execute the rollback command to cancel the maintenance and restore 'ready' state","Check execution_payload_state (SELECT operation FROM execution_payload_state) to identify the stuck operation and consult rotation tooling logs to finish it","Avoid starting nodes concurrently with key-rotation maintenance; serialize operations via a maintenance window"],"exampleFix":"// before: node started while rewrap pending, fails\nensure_execution_payload_storage(&keys).await?;\n// after: finish the pending rewrap first, then connect\nbegin_execution_payload_rewrap(&keys).await?; // resumes/completes rewrap to 'ready'\nensure_execution_payload_storage(&keys).await?;","handlingStrategy":"validation","validationCode":"let op: Option<String> = sqlx::query_scalar(\n    \"SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions'\")\n    .fetch_optional(&pool).await?;\nif let Some(op) = op {\n    anyhow::ensure!(op == \"ready\" || op == \"migrate\", \"payload maintenance '{op}' pending; finish it before connecting\");\n}","typeGuard":"fn storage_ready(operation: &str) -> bool {\n    matches!(operation, \"ready\" | \"migrate\")\n}","tryCatchPattern":"match db.ensure_execution_payload_storage(&keys).await {\n    Err(e) if e.to_string().contains(\"maintenance; complete or roll back\") => {\n        // run the matching rotation/rollback tool to reach 'ready', then reconnect\n    }\n    r => r?,\n}","preventionTips":["Check execution_payload_state.operation before node startup during maintenance windows","Never kill key-rotation or rollback jobs mid-run; resume them to completion","Serialize all payload maintenance through a single operator/tool","Alert on state rows left non-ready beyond a threshold"],"tags":["rust","database","encryption","maintenance","key-rotation"],"backgroundTag":"invalid-state-transition","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}