nautechsystems/nautilus_trader · error

Execution payload storage is in {operation} maintenance; com

Error message

Execution payload storage is in {operation} maintenance; complete or roll back that operation before connecting

What it means

ensure_execution_payload_storage connects to protected signed-transaction storage and processes any pending maintenance operation recorded in execution_payload_state. When the stored operation is neither 'migrate' nor 'ready' (e.g. 'rewrap' or 'rollback'), it refuses to connect because the storage schema is mid-transition and must be completed or rolled back first. This is a safety gate so connections never observe half-encrypted payload rows.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:4638

            ),
        }

        loop {
            let state = self.execution_payload_state().await?.ok_or_else(|| {
                anyhow::anyhow!("Execution payload marker exists without durable state")
            })?;
            validate_execution_payload_state(&state, keys)?;
            match state.operation.as_str() {
                "migrate" => {
                    if self
                        .migrate_execution_payload_batch(keys, EXECUTION_PAYLOAD_BATCH_SIZE)
                        .await?
                    {
                        break;
                    }
                }
                "ready" => break,
                operation => anyhow::bail!(
                    "Execution payload storage is in {operation} maintenance; complete or roll back that operation before connecting"
                ),
            }
        }

        self.validate_execution_payload_ready(keys).await
    }

    /// Requires ready protected storage and authenticates every persisted signed payload.
    pub(crate) async fn require_execution_payload_storage(
        &self,
        keys: &PayloadKeySet,
        policy: PayloadPolicy,
        batch_size: i64,
    ) -> anyhow::Result<ExecutionPayloadLease> {
        self.require_execution_payload_storage_ready(keys).await?;
        let (check, transaction) = self
            .inspect_execution_payload_storage(Some(keys), Some(policy), batch_size)

View on GitHub (pinned to 18893faf8b)

Solutions

1. Resume/complete the in-progress operation by running the corresponding rotation/rollback command (rewrap_execution_payload or rollback path) until state.operation becomes 'ready'
2. Alternatively execute the rollback command to cancel the maintenance and restore 'ready' state
3. Check execution_payload_state (SELECT operation FROM execution_payload_state) to identify the stuck operation and consult rotation tooling logs to finish it
4. Avoid starting nodes concurrently with key-rotation maintenance; serialize operations via a maintenance window

Example fix

// before: node started while rewrap pending, fails
ensure_execution_payload_storage(&keys).await?;
// after: finish the pending rewrap first, then connect
begin_execution_payload_rewrap(&keys).await?; // resumes/completes rewrap to 'ready'
ensure_execution_payload_storage(&keys).await?;

let op: Option<String> = sqlx::query_scalar(
    "SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions'")
    .fetch_optional(&pool).await?;
if let Some(op) = op {
    anyhow::ensure!(op == "ready" || op == "migrate", "payload maintenance '{op}' pending; finish it before connecting");
}

fn storage_ready(operation: &str) -> bool {
    matches!(operation, "ready" | "migrate")
}

match db.ensure_execution_payload_storage(&keys).await {
    Err(e) if e.to_string().contains("maintenance; complete or roll back") => {
        // run the matching rotation/rollback tool to reach 'ready', then reconnect
    }
    r => r?,
}

When it happens

Trigger: Calling ensure_execution_payload_storage (normal node startup/connect path) while execution_payload_state.operation holds a maintenance value such as 'rewrap' or 'rollback' left over from an interrupted or in-progress key rotation or protection rollback.

Common situations: A key-rotation (rewrap) or rollback run crashed or was killed midway and its state row never returned to 'ready'; a second node instance started while another operator was running rotation tooling; manual maintenance was started via admin scripts and forgotten.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Execution payload storage is in {operation} maintenance, not
• Execution payload storage is in {operation} maintenance, not
• Execution payload protection is active, but no payload key i
• Protected execution transaction {} contains plaintext
• Protected execution transaction {} has no envelope

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/4fef2359897199a9. Report an issue: GitHub.