nautechsystems/nautilus_trader · error · anyhow::Error

Failed to extend finalized header ledger: {e}

Error message

Failed to extend finalized header ledger: {e}

What it means

This error wraps a sqlx/PostgreSQL failure that occurs while inserting a subsequent verified finalized header (all headers after the first) into `execution_verified_finalized_header` during ledger extension. The insert uses `ON CONFLICT (chain_id, wallet_address, number) DO NOTHING`, so it fails only on genuine database-level errors, not on duplicate rows. The original driver error is preserved in the message via `{e}` and the operation runs inside a transaction, so any failure rolls back the whole bootstrap.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:4484

                INSERT INTO execution_verified_finalized_header (
                    chain_id, wallet_address, number, hash, parent_hash, timestamp,
                    base_fee_per_gas, manifest_digest
                )
                VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
                ON CONFLICT (chain_id, wallet_address, number) DO NOTHING
                ",
            )
            .bind(chain_id)
            .bind(bootstrap.wallet_address)
            .bind(number)
            .bind(&header.hash)
            .bind(&header.parent_hash)
            .bind(timestamp)
            .bind(&base_fee)
            .bind(bootstrap.manifest_digest)
            .execute(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to extend finalized header ledger: {e}"))?;
            let stored = sqlx::query_as::<_, (String, String, i64, Option<String>, String)>(
                "
                SELECT hash, parent_hash, timestamp, base_fee_per_gas, manifest_digest
                FROM execution_verified_finalized_header
                WHERE chain_id = $1 AND wallet_address = $2 AND number = $3
                ",
            )
            .bind(chain_id)
            .bind(bootstrap.wallet_address)
            .bind(number)
            .fetch_one(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to validate finalized header ledger: {e}"))?;
            anyhow::ensure!(
                stored
                    == (
                        header.hash.clone(),
                        header.parent_hash.clone(),

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Read the wrapped `{e}` cause to identify the concrete sqlx/Postgres error (connection, constraint, timeout) before changing anything.
  2. Check database connectivity, pool limits, and statement/idle-in-transaction timeouts; retry the bootstrap on transient connection errors.
  3. Compare the running schema against the expected `execution_verified_finalized_header` definition and apply pending migrations after version upgrades.
  4. Reduce batch size or chunk the insert if statement_timeout is hit on large finalized ranges.
  5. Inspect PostgreSQL server logs at failure time for locks, deadlocks, or failover events; ensure a single writer owns the bootstrap transaction.

Example fix

// before: one huge transaction for the entire finalized range
let mut tx = pool.begin().await?;
for h in headers { insert_header(&mut tx, h).await?; }

// after: bounded chunks with retry on transient errors
for chunk in headers.chunks(500) {
    let mut tx = pool.begin().await?;
    for h in chunk { insert_header(&mut tx, h).await?; }
    tx.commit().await?; // smaller statements, shorter lock hold
}
Defensive patterns

Strategy: retry

Validate before calling

let reachable = sqlx::query("SELECT 1").execute(&pool).await.is_ok();
if !reachable { return Err(anyhow!("database unavailable before bootstrap")); }
let schema_ok = sqlx::query(
    "SELECT 1 FROM information_schema.tables WHERE table_name = 'execution_verified_finalized_header'",
).fetch_optional(&pool).await?.is_some();
if !schema_ok { return Err(anyhow!("ledger table missing; run migrations")); }

Try / catch

match insert_finalized_headers(&mut tx, &headers).await {
    Err(e) if is_transient(&e) => backoff_retry(|| insert_finalized_headers(&mut tx, &headers), 3).await?,
    Err(e) if e.to_string().contains("Failed to extend finalized header ledger") => {
        tracing::error!(cause = %e, "ledger extension failed");
        return Err(e);
    }
    Ok(()) => {}
}

Prevention

When it happens

Trigger: Any PostgreSQL error during the INSERT for headers 2..n of `bootstrap.finalized_headers`: connection dropped mid-transaction, constraint violation other than the ignored unique conflict (e.g. NOT NULL or type errors from oversized/malformed hash or base_fee strings), lock timeout on the ledger rows, statement timeout, or the table/sequence being unavailable or migrated concurrently.

Common situations: Network blips between the adapter and PostgreSQL during long bootstrap transactions; schema drift after an upgrade (columns/added NOT NULL constraints); exceeding statement_timeout on very large finalized batches; database failover; connection pool exhaustion under concurrent bootstraps.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/dc14126daab1c692. Report an issue: GitHub.