nautechsystems/nautilus_trader · error · anyhow::Error

Execution payload rewrap left {remaining} row(s)

Error message

Execution payload rewrap left {remaining} row(s)

What it means

When a rewrap batch finds no more rows to process, the code locks the table and runs a verification COUNT of rows that still have `payload_expected` with a NULL/missing plaintext constraint violation: raw_transaction set, sealed_transaction NULL, or an envelope not sealed under the new active key. If any rows remain (`remaining != 0`), the rewrap is not actually complete and this error aborts the state transition to 'ready'. It is a safety check that the batch loop truly finished the rewrap.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5470

        .await
        .context("failed to load execution payload rewrap batch")?;

        if rows.is_empty() {
            sqlx::query("LOCK TABLE execution_transaction_hash IN SHARE ROW EXCLUSIVE MODE")
                .execute(&mut *transaction)
                .await
                .context("failed to lock execution payload rewrap completion")?;
            let remaining = sqlx::query_scalar::<_, i64>(
                "SELECT COUNT(*) FROM execution_transaction_hash \
                 WHERE payload_expected \
                   AND (raw_transaction IS NOT NULL OR sealed_transaction IS NULL \
                        OR substring(sealed_transaction FROM 2 FOR 32) <> $1)",
            )
            .bind(keys.active_key_id().as_slice())
            .fetch_one(&mut *transaction)
            .await
            .context("failed to verify execution payload rewrap completion")?;
            anyhow::ensure!(
                remaining == 0,
                "Execution payload rewrap left {remaining} row(s)"
            );
            sqlx::query(
                "UPDATE execution_payload_state SET operation = 'ready', updated_at = NOW() \
                 WHERE component = 'signed_transactions' AND operation = 'rewrap'",
            )
            .execute(&mut *transaction)
            .await
            .context("failed to mark execution payload rewrap complete")?;
            transaction
                .commit()
                .await
                .context("failed to commit execution payload rewrap completion")?;
            return Ok(true);
        }

        for hash in rows {

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Identify the leftover rows: `SELECT id FROM execution_transaction_hash WHERE payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NULL OR substring(sealed_transaction FROM 2 FOR 32) <> $1)`.
  2. Stop all writers to `execution_transaction_hash` (adapter instances, ETL jobs) and re-run the rewrap so the batch loop can process the remaining rows.
  3. Fix half-migrated rows: either re-seal them under the active key (restore from a source with plaintext + re-seal) or correct `payload_expected` if these rows should not carry payloads.
  4. Check for concurrent insert paths and ensure they seal payloads with the current active key before the rewrap declares completion.
Defensive patterns

Strategy: retry

Validate before calling

let remaining: i64 = sqlx::query_scalar("SELECT COUNT(*) FROM execution_transaction_hash WHERE payload_expected AND (raw_transaction IS NOT NULL OR sealed_transaction IS NULL OR substring(sealed_transaction FROM 2 FOR 32) <> $1)")
    .bind(active_key_id.as_slice()).fetch_one(&mut *tx).await?;
// only declare the rewrap complete when remaining == 0

Try / catch

match rewrap_result {
    Ok(done) if done => {/* rewrap complete */},
    Err(e) if e.to_string().contains("Execution payload rewrap left") => {
        // quiesce writers, then retry the batch loop after fixing/remediating rows
    },
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: Rows exist that the batch SELECT (which filters on `substring(sealed_transaction FROM 2 FOR 32) <> $1` only for non-null envelopes) skipped but the stricter verification COUNT flags — e.g. rows with `payload_expected=true` but `sealed_transaction IS NULL` or `raw_transaction IS NOT NULL` (plaintext rows), or rows inserted concurrently between the last batch and the table lock on a table where the lock was not held by a competing writer.

Common situations: A writer inserted new signed-transaction rows with plaintext `raw_transaction` while a rewrap was running; a previous failed rewrap left rows half-migrated (NULL envelopes); manual data loads backfilled `payload_expected` rows without sealed envelopes; trigger/ETL jobs writing to `execution_transaction_hash` during maintenance.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/0349e7c136973899. Report an issue: GitHub.