What causes "Execution payload {} changed during migration" in nautechsystems/nautilus_trader?

Ensure only one migrator process runs at a time (advisory lock or single-instance deployment)

nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload {} changed during migration

Error message

Execution payload {} changed during migration

What it means

This error is thrown by the execution payload migration path when an UPDATE intended to promote exactly one execution payload row affected zero rows. It means the row for the given hash id was changed or deleted concurrently between the SELECT and the UPDATE inside the transaction, so the migration aborted to avoid silently losing data.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:4987

            let unsealed = keys.unseal(&envelope, &context)?;
            authenticate_retained_payload(&unsealed, &intent, &hash, keys.deployment_id())?;
            anyhow::ensure!(
                unsealed == raw_transaction,
                "Execution payload {} changed during seal round trip",
                hash.id
            );
            let result = sqlx::query(
                "UPDATE execution_transaction_hash \
                 SET sealed_transaction = $2, raw_transaction = NULL, updated_at = NOW() \
                 WHERE id = $1 AND raw_transaction = $3 AND sealed_transaction IS NULL",
            )
            .bind(hash.id)
            .bind(&envelope)
            .bind(raw_transaction)
            .execute(&mut *transaction)
            .await
            .context("failed to promote execution payload")?;
            anyhow::ensure!(
                result.rows_affected() == 1,
                "Execution payload {} changed during migration",
                hash.id
            );
            sqlx::query(
                "UPDATE execution_payload_state SET progress_id = $1, updated_at = NOW() \
                 WHERE component = 'signed_transactions'",
            )
            .bind(hash.id)
            .execute(&mut *transaction)
            .await
            .context("failed to record execution payload migration progress")?;
        }

        transaction
            .commit()
            .await
            .context("failed to commit execution payload migration batch")?;

View on GitHub (pinned to 18893faf8b)

Solutions

Ensure only one migrator process runs at a time (advisory lock or single-instance deployment)
Re-run the migration; it is transactional and aborted atomically, so state should be consistent
Check for concurrent writers/deleters on execution_payload tables and pause them during migration
Verify the hash id still exists before migrating

Example fix

// before: blind concurrent migration
let result = sqlx::query("UPDATE ... ").bind(hash.id).execute(&mut *transaction).await?;
anyhow::ensure!(result.rows_affected() == 1, "Execution payload {} changed during migration", hash.id);
// after: take an advisory lock first
sqlx::query("SELECT pg_advisory_xact_lock($1)").bind(MIGRATION_LOCK_KEY).execute(&mut *transaction).await?;

Defensive patterns

Strategy: try-catch

Validate before calling

let exists: Option<i64> = sqlx::query_scalar("SELECT 1 FROM execution_payload WHERE id = $1").bind(hash.id).fetch_optional(&mut conn).await?;
if exists.is_none() { return Err(anyhow::anyhow!("payload {} absent before migration", hash.id)); }

Try / catch

match migrate_payload(&mut tx, hash).await {
    Err(e) if e.to_string().contains("changed during migration") => {
        // serialize migrations: take advisory lock and retry once
        sqlx::query("SELECT pg_advisory_xact_lock($1)").bind(LOCK_KEY).execute(&mut tx).await?;
        retry_migration(hash).await
    }
    other => other,
}

Prevention

Run migrations single-instance or under an advisory lock
Never mutate execution payload tables outside the migration path
Re-run the transactional migration after an abort — it commits atomically
Monitor for concurrent writers during migration windows

When it happens

Trigger: Calling the migration routine while another transaction deletes or rewrites the execution_payload row with the same hash id; running migration against a database where the expected row was already consumed by a prior partial migration; concurrent migration processes racing on the same hash.

Common situations: Two instances of the node/migrator running against the same database at once; a manual DBA cleanup removing rows mid-migration; re-running a failed migration whose transaction partially committed elsewhere.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/bc01da7772078125. Report an issue: GitHub.