nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload storage is not rolling back

Error message

Execution payload storage is not rolling back

What it means

This error comes from the batched execution-payload rollback in the blockchain cache database. Each batch transaction re-reads the execution_payload_state row (locked FOR UPDATE) and asserts its operation column equals 'rollback'. If the state row shows anything other than 'rollback', the storage layer has concurrently moved out of rollback mode (e.g. back to 'ready' or into another maintenance operation), and continuing would corrupt the two-representation (raw/sealed) payload migration.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5633

        batch_size: i64,
    ) -> anyhow::Result<bool> {
        let mut transaction = self
            .pool
            .begin()
            .await
            .context("failed to start execution payload rollback batch")?;
        lock_execution_payload_operation(&mut transaction).await?;
        let state_row = sqlx::query(
            "SELECT deployment_id, protocol_version, operation, active_key_id \
             FROM execution_payload_state WHERE component = 'signed_transactions' FOR UPDATE",
        )
        .fetch_optional(&mut *transaction)
        .await
        .context("failed to lock execution payload rollback state")?
        .ok_or_else(|| anyhow::anyhow!("Execution payload rollback state is missing"))?;
        let state = execution_payload_state_from_row(&state_row)?;
        validate_execution_payload_state(&state, keys)?;
        anyhow::ensure!(
            state.operation == "rollback",
            "Execution payload storage is not rolling back"
        );
        let rows = sqlx::query_as::<_, ExecutionTransactionHashRow>(
            "
            SELECT
                id, intent_id, chain_id, transaction_hash, payload_expected,
                raw_transaction, sealed_transaction, status, block_number, block_hash,
                receipt_success, gas_used, effective_gas_price, current
            FROM execution_transaction_hash
            WHERE payload_expected AND sealed_transaction IS NOT NULL
            ORDER BY id
            LIMIT $1
            FOR UPDATE
            ",
        )
        .bind(batch_size)
        .fetch_all(&mut *transaction)

View on GitHub (pinned to 18893faf8b)

Solutions

1. Check the execution_payload_state table (component='signed_transactions') to see the current operation value; ensure only one maintenance workflow is running.
2. Take an advisory/application-level lock so seal and rollback maintenance cannot run concurrently, then retry the rollback.
3. If the rollback actually completed elsewhere, do not resume; verify row consistency and let the normal path proceed.
4. If the state was manually modified, restore operation='rollback' (or restart the rollback from a consistent 'ready' state) before retrying.

Example fix

// before: two services racing
// service A: db.rollback_execution_payload(&keys, 1000).await?;
// service B (concurrently): db.seal_execution_payload(...).await?;

// after: serialize maintenance with an advisory lock
let mut conn = db.pool.acquire().await?;
sqlx::query("SELECT pg_advisory_lock(hashtext('execution_payload_maintenance'))")
    .execute(&mut *conn).await?;
db.rollback_execution_payload(&keys, 1000).await?;
sqlx::query("SELECT pg_advisory_unlock(hashtext('execution_payload_maintenance'))")
    .execute(&mut *conn).await?;

let state: (String,) = sqlx::query_as(
    "SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions'",
).fetch_one(&mut conn).await?;
if state.0 != "rollback" && state.0 != "ready" {
    return Err(anyhow!("payload storage in '{}' maintenance; rollback not safe", state.0));
}

match db.rollback_execution_payload(&keys, batch).await {
    Ok(()) => {},
    Err(e) if e.to_string().contains("not rolling back") => {
        // another maintenance operation holds/changed the state; serialize and retry later
    },
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling rollback_execution_payload (which begins a rollback then loops rollback_execution_payload_batch) while another process concurrently completes, re-initializes, or re-enters the payload protection workflow, changing execution_payload_state.operation from 'rollback' to something else between the begin and batch phases. Also occurs if the state row was manually edited or a maintenance/seal operation was started mid-rollback.

Common situations: Two operators or two application instances running payload seal/rollback maintenance at the same time against the same database; a manual SQL fix-up of execution_payload_state during an in-flight rollback; resuming a rollback after another host already finished it and reset state; stale orchestration scripts racing the app's own migration job.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Execution payload {} changed during rollback
• Execution payload {} changed while clearing rollback envelop
• Execution payload rollback left {invalid} invalid row(s)
• Execution payload {} contains both representations during ro
• Execution intent {intent_id} is not prepared for nonce {nonc

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/a713f591a4331026. Report an issue: GitHub.