What causes "Execution payload storage is in {} maintenance" in nautechsystems/nautilus_trader?

Wait for the in-flight maintenance operation to complete, then retry the lease acquisition

nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload storage is in {} maintenance

Error message

Execution payload storage is in {} maintenance

What it means

Execution payload storage has a lifecycle operation column; leases are only granted when operation == 'ready'. While any other maintenance operation (e.g. activate, migrate, seal, rotate) is recorded, acquire_execution_payload_lease refuses to proceed so readers/writers never observe a half-migrated store.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:4715

        let version = marker.ok_or_else(|| {
            anyhow::anyhow!("Postgres execution requires protected payload storage")
        })?;
        anyhow::ensure!(
            version == EXECUTION_PAYLOAD_PROTOCOL_VERSION,
            "Unsupported execution payload protection version {version}"
        );
        let row = sqlx::query(
            "SELECT deployment_id, protocol_version, operation, active_key_id \
             FROM execution_payload_state WHERE component = 'signed_transactions' \
             FOR SHARE",
        )
        .fetch_optional(&mut *transaction)
        .await
        .context("failed to lock execution payload state")?
        .ok_or_else(|| anyhow::anyhow!("Execution payload marker exists without state"))?;
        let state = execution_payload_state_from_row(&row)?;
        validate_execution_payload_state(&state, keys)?;
        anyhow::ensure!(
            state.operation == "ready",
            "Execution payload storage is in {} maintenance",
            state.operation
        );

        Ok(ExecutionPayloadLease {
            _transaction: transaction,
        })
    }

    /// Reserves one nonce use under the active payload key.
    pub(crate) async fn reserve_execution_payload_seal(
        &self,
        keys: &PayloadKeySet,
    ) -> anyhow::Result<()> {
        let mut transaction = self
            .pool
            .begin()

View on GitHub (pinned to 18893faf8b)

Solutions

Wait for the in-flight maintenance operation to complete, then retry the lease acquisition
If a migration is stuck (crashed mid-run), resume or finalize it via migrate_execution_payload_batch until operation returns to 'ready'
Inspect SELECT operation FROM execution_payload_state WHERE component='signed_transactions' to identify the current operation
Coordinate maintenance windows so signing/broadcast does not run concurrently with migrations

Example fix

// before: leasing during migration
let lease = db.acquire_execution_payload_lease(&keys).await?;
// after: check operation first
if db.execution_payload_state().await?.map(|s| s.operation) != Some("ready".into()) {
    // defer signing until maintenance completes
}

Defensive patterns

Strategy: retry

Validate before calling

let op: Option<String> = sqlx::query_scalar(
    "SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions'",
).fetch_optional(&pool).await?;
let ready = op.as_deref() == Some("ready");

Try / catch

loop {
    match db.acquire_execution_payload_lease(&keys).await {
        Ok(lease) => break lease,
        Err(e) if e.to_string().contains("maintenance") => {
            tokio::time::sleep(Duration::from_secs(5)).await; // wait out maintenance
        }
        Err(e) => return Err(e),
    }
}

Prevention

Schedule migrations/rotations outside trading windows
Check storage operation status before starting signing workers
Always finalize migrations (drive batches until they report completion)
Alert when operation stays non-ready beyond an expected duration

When it happens

Trigger: Calling acquire_execution_payload_lease while execution_payload_state.operation for 'signed_transactions' is anything other than 'ready' — typically during an in-flight migration, key rotation, or activation initiated elsewhere.

Common situations: A second node/process attempts signing while a migration is running; a previous migration crashed leaving operation stuck at a non-ready value; key rotation scheduled concurrently with trading.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/6b6abc18dad40c57. Report an issue: GitHub.