nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload {} changed during rollback

Error message

Execution payload {} changed during rollback

What it means

After unsealing the envelope and writing the plaintext raw_transaction, the code verifies exactly one row was affected by an UPDATE guarded on (id, raw_transaction IS NULL, sealed_transaction = envelope). Zero affected rows means the row changed underneath the rollback — the expected snapshot no longer matches — so the batch aborts to avoid overwriting a concurrent modification with a stale unsealed payload.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5714

            let envelope = hash
                .sealed_transaction
                .as_deref()
                .expect("rollback query requires envelope");
            let intent = load_execution_intent(&mut transaction, hash.intent_id).await?;
            let context = payload_context(&intent, &hash, keys.deployment_id())?;
            let raw_transaction = keys.unseal(envelope, &context)?;
            authenticate_retained_payload(&raw_transaction, &intent, &hash, keys.deployment_id())?;
            let result = sqlx::query(
                "UPDATE execution_transaction_hash SET raw_transaction = $2, updated_at = NOW() \
                 WHERE id = $1 AND raw_transaction IS NULL AND sealed_transaction = $3",
            )
            .bind(hash.id)
            .bind(&raw_transaction)
            .bind(envelope)
            .execute(&mut *transaction)
            .await
            .context("failed to recreate plaintext execution payload")?;
            anyhow::ensure!(
                result.rows_affected() == 1,
                "Execution payload {} changed during rollback",
                hash.id
            );
            authenticate_retained_payload(&raw_transaction, &intent, &hash, keys.deployment_id())?;
            let result = sqlx::query(
                "UPDATE execution_transaction_hash SET sealed_transaction = NULL, updated_at = NOW() \
                 WHERE id = $1 AND raw_transaction = $2 AND sealed_transaction = $3",
            )
            .bind(hash.id)
            .bind(&raw_transaction)
            .bind(envelope)
            .execute(&mut *transaction)
            .await
            .context("failed to clear rolled-back execution payload envelope")?;
            anyhow::ensure!(
                result.rows_affected() == 1,
                "Execution payload {} changed while clearing rollback envelope",

View on GitHub (pinned to 18893faf8b)

Solutions

1. Stop all other writers/maintenance on execution_transaction_hash while the rollback runs.
2. Verify the row (id from the message): confirm its current sealed_transaction/raw_transaction values and whether another process altered them.
3. Rerun the rollback from the current state once concurrent writers are stopped; the FOR UPDATE re-read will pick up the new snapshot.
4. Ensure all maintenance paths acquire the execution payload operation lock (lock_execution_payload_operation) before touching these rows.

Example fix

// before: concurrent write racing the rollback UPDATE
// some_tool: UPDATE execution_transaction_hash SET sealed_transaction=$new WHERE id=$id; -- no lock
// rollback: rows_affected == 0 -> "Execution payload {id} changed during rollback"

// after: hold the maintenance lock for the whole window
// SELECT pg_advisory_lock(hashtext('execution_payload_maintenance'));
// ... run rollback_execution_payload ... 
// SELECT pg_advisory_unlock(hashtext('execution_payload_maintenance'));

let locked: (String,) = sqlx::query_as(
    "SELECT operation FROM execution_payload_state WHERE component = 'signed_transactions' FOR UPDATE",
).fetch_optional(&mut conn).await?.ok_or_else(|| anyhow!("payload protection not active"))?;
// proceed only while holding the maintenance advisory lock and state is 'rollback'

match db.rollback_execution_payload(&keys, batch).await {
    Err(e) if e.to_string().contains("changed during rollback") => {
        // stop concurrent writers, then retry the rollback once
        db.rollback_execution_payload(&keys, batch).await?;
    },
    other => other?,
}

When it happens

Trigger: Between the SELECT ... FOR UPDATE batch read and the UPDATE, the row's sealed_transaction or raw_transaction was modified by another connection (possible if lock ordering with the fence trigger/advisory lock was bypassed), or a manual edit cleared/moved the payload. Rows_affected == 0 on the guarded recreate-plaintext UPDATE throws this error.

Common situations: Another node or an operator running conflicting maintenance without the payload operation lock; manual SQL updates to execution_transaction_hash during rollback; an application bug writing transaction payloads concurrently with a rollback window.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Execution payload {} changed while clearing rollback envelop
• Execution payload storage is not rolling back
• Execution payload rollback left {invalid} invalid row(s)
• Execution payload {} contains both representations during ro
• Failed to lock execution intent {intent_id}: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/cee132ee36013b92. Report an issue: GitHub.