nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload {} changed while clearing rollback envelop

Error message

Execution payload {} changed while clearing rollback envelope

What it means

This is the second guard of the per-row rollback pair: after recreating the plaintext raw_transaction, the code clears sealed_transaction with an UPDATE guarded on (id, raw_transaction = unsealed value, sealed_transaction = envelope). If rows_affected != 1, the row changed between the two updates (raw value or envelope no longer match), so the batch aborts to prevent leaving the row in a double-representation or payload-less state.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5730

            .await
            .context("failed to recreate plaintext execution payload")?;
            anyhow::ensure!(
                result.rows_affected() == 1,
                "Execution payload {} changed during rollback",
                hash.id
            );
            authenticate_retained_payload(&raw_transaction, &intent, &hash, keys.deployment_id())?;
            let result = sqlx::query(
                "UPDATE execution_transaction_hash SET sealed_transaction = NULL, updated_at = NOW() \
                 WHERE id = $1 AND raw_transaction = $2 AND sealed_transaction = $3",
            )
            .bind(hash.id)
            .bind(&raw_transaction)
            .bind(envelope)
            .execute(&mut *transaction)
            .await
            .context("failed to clear rolled-back execution payload envelope")?;
            anyhow::ensure!(
                result.rows_affected() == 1,
                "Execution payload {} changed while clearing rollback envelope",
                hash.id
            );
            sqlx::query(
                "UPDATE execution_payload_state SET progress_id = $1, updated_at = NOW() \
                 WHERE component = 'signed_transactions'",
            )
            .bind(hash.id)
            .execute(&mut *transaction)
            .await
            .context("failed to record execution payload rollback progress")?;
        }
        transaction
            .commit()
            .await
            .context("failed to commit execution payload rollback batch")?;
        Ok(false)

View on GitHub (pinned to 18893faf8b)

Solutions

1. Stop concurrent writers to execution_transaction_hash for the duration of the rollback and rerun it.
2. Inspect the row (id from the message) for the actual current state and repair to a single consistent representation before retrying.
3. Ensure every maintenance path takes the execution payload operation lock so seal/rollback steps never interleave.
4. Check the execution_transaction_payload_fence trigger is intact during rollback; dropping it early lets unsafe concurrent writes through.

Example fix

-- diagnose
SELECT id, raw_transaction IS NOT NULL AS has_raw, sealed_transaction IS NOT NULL AS has_sealed
FROM execution_transaction_hash WHERE id = $offending_id;

// before: interleaved writers corrupt the two-step rollback
// writer: UPDATE execution_transaction_hash SET raw_transaction=$other WHERE id=$id;
// rollback clear-envelope UPDATE matches 0 rows -> error

// after: serialize maintenance with the operation lock before retrying
// SELECT pg_advisory_lock(hashtext('execution_payload_maintenance'));
// db.rollback_execution_payload(&keys, 1000).await?;
// SELECT pg_advisory_unlock(hashtext('execution_payload_maintenance'));

let mid: Vec<(i64,)> = sqlx::query_as(
    "SELECT id FROM execution_transaction_hash \
     WHERE payload_expected AND sealed_transaction IS NOT NULL AND raw_transaction IS NOT NULL",
).fetch_all(&mut conn).await?;
if !mid.is_empty() {
    return Err(anyhow!("rollback interrupted mid-pair on rows {:?}; resolve first", mid));
}

match db.rollback_execution_payload(&keys, batch).await {
    Err(e) if e.to_string().contains("changed while clearing rollback envelope") => {
        // acquire exclusive maintenance lock, verify row state, retry rollback
    },
    other => other?,
}

When it happens

Trigger: A concurrent writer modified raw_transaction or sealed_transaction for the row between the recreate-plaintext UPDATE and the clear-envelope UPDATE, or an external process cleared the envelope first. The strict guarded UPDATE then affects 0 rows and the ensure fires.

Common situations: Another application instance running overlapping maintenance without the shared operation lock; manual fixes applied mid-rollback; trigger-based tooling rewriting payload columns during the rollback window.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Execution payload {} changed during rollback
• Execution payload storage is not rolling back
• Execution payload rollback left {invalid} invalid row(s)
• Execution payload {} contains both representations during ro
• Failed to lock execution intent {intent_id}: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/6c2bd0ca11ee6d30. Report an issue: GitHub.