nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload rollback left {invalid} invalid row(s)

Error message

Execution payload rollback left {invalid} invalid row(s)

What it means

When the rollback batch finds no remaining sealed rows, it locks the whole execution_transaction_hash table and counts rows whose payload representation contradicts their payload_expected flag (raw/sealed present or missing in the wrong combination). If that count is nonzero, the rollback is incomplete or inconsistent, so it refuses to drop the fence trigger and protection state, throwing this error instead of committing a partially converted table.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5668

        .bind(batch_size)
        .fetch_all(&mut *transaction)
        .await
        .context("failed to load execution payload rollback batch")?;

        if rows.is_empty() {
            sqlx::query("LOCK TABLE execution_transaction_hash IN SHARE ROW EXCLUSIVE MODE")
                .execute(&mut *transaction)
                .await
                .context("failed to lock execution payload rollback completion")?;
            let invalid = sqlx::query_scalar::<_, i64>(
                "SELECT COUNT(*) FROM execution_transaction_hash \
                 WHERE (payload_expected AND (raw_transaction IS NULL OR sealed_transaction IS NOT NULL)) \
                    OR (NOT payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NOT NULL))",
            )
            .fetch_one(&mut *transaction)
            .await
            .context("failed to verify execution payload rollback completion")?;
            anyhow::ensure!(
                invalid == 0,
                "Execution payload rollback left {invalid} invalid row(s)"
            );

            for statement in [
                "DROP TRIGGER IF EXISTS execution_transaction_payload_fence ON execution_transaction_hash",
                "DELETE FROM execution_payload_state WHERE component = 'signed_transactions'",
                "DELETE FROM execution_schema_version WHERE component = 'evm_execution_payload'",
            ] {
                sqlx::query(statement)
                    .execute(&mut *transaction)
                    .await
                    .context("failed to complete execution payload rollback")?;
            }
            transaction
                .commit()
                .await
                .context("failed to commit execution payload rollback completion")?;

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Run the same COUNT query to identify the offending rows: SELECT id FROM execution_transaction_hash WHERE (payload_expected AND (raw_transaction IS NULL OR sealed_transaction IS NOT NULL)) OR (NOT payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NOT NULL)).
  2. Fix the inconsistent rows (backfill raw_transaction for payload_expected rows, or clear payloads for rows not expecting them) so each row matches its payload_expected flag.
  3. Ensure all writers are stopped or using a schema version consistent with the rollback before retrying rollback_execution_payload.
  4. If rows are unrecoverable, quarantine or delete them deliberately, then rerun the rollback to completion.

Example fix

-- diagnose
SELECT id, payload_expected, raw_transaction IS NULL AS no_raw,
       sealed_transaction IS NOT NULL AS has_sealed
FROM execution_transaction_hash
WHERE (payload_expected AND (raw_transaction IS NULL OR sealed_transaction IS NOT NULL))
   OR (NOT payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NOT NULL));

// before: retrying rollback with inconsistent rows
// db.rollback_execution_payload(&keys, 1000).await?; // fails with "left N invalid row(s)"
// after: repair rows first, then retry
// UPDATE execution_transaction_hash SET sealed_transaction = NULL WHERE id = $bad_id; -- after restoring raw_transaction
Defensive patterns

Strategy: validation

Validate before calling

let invalid: i64 = sqlx::query_scalar(
    "SELECT COUNT(*) FROM execution_transaction_hash \
     WHERE (payload_expected AND (raw_transaction IS NULL OR sealed_transaction IS NOT NULL)) \
        OR (NOT payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NOT NULL))",
).fetch_one(&mut conn).await?;
if invalid != 0 {
    return Err(anyhow!("{invalid} payload rows inconsistent; repair before rollback"));
}

Try / catch

match db.rollback_execution_payload(&keys, batch).await {
    Err(e) if e.to_string().contains("invalid row(s)") => {
        // run the diagnostic COUNT query, repair rows, retry once
    },
    other => other?,
}

Prevention

When it happens

Trigger: Completing the final rollback batch while some rows still have raw_transaction IS NULL despite payload_expected, sealed_transaction NOT NULL, or rows with payloads despite payload_expected=false — i.e. rows modified outside the rollback loop, inserted during rollback with wrong representation, or corrupted by manual edits.

Common situations: Rows inserted by a concurrent writer (with a stale trigger/fence) during a long rollback; manual data fixes that set raw_transaction or sealed_transaction inconsistently; restoring the table from a partial backup; a previous failed migration that left mixed-representation rows.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/0a97af296c6066a2. Report an issue: GitHub.