{"record":{"id":"dc14126daab1c692","repo":"nautechsystems/nautilus_trader","slug":"failed-to-extend-finalized-header-ledger-e","errorCode":null,"errorMessage":"Failed to extend finalized header ledger: {e}","messagePattern":"Failed to extend finalized header ledger: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":4484,"sourceCode":"                INSERT INTO execution_verified_finalized_header (\n                    chain_id, wallet_address, number, hash, parent_hash, timestamp,\n                    base_fee_per_gas, manifest_digest\n                )\n                VALUES ($1, $2, $3, $4, $5, $6, $7, $8)\n                ON CONFLICT (chain_id, wallet_address, number) DO NOTHING\n                \",\n            )\n            .bind(chain_id)\n            .bind(bootstrap.wallet_address)\n            .bind(number)\n            .bind(&header.hash)\n            .bind(&header.parent_hash)\n            .bind(timestamp)\n            .bind(&base_fee)\n            .bind(bootstrap.manifest_digest)\n            .execute(&mut *transaction)\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to extend finalized header ledger: {e}\"))?;\n            let stored = sqlx::query_as::<_, (String, String, i64, Option<String>, String)>(\n                \"\n                SELECT hash, parent_hash, timestamp, base_fee_per_gas, manifest_digest\n                FROM execution_verified_finalized_header\n                WHERE chain_id = $1 AND wallet_address = $2 AND number = $3\n                \",\n            )\n            .bind(chain_id)\n            .bind(bootstrap.wallet_address)\n            .bind(number)\n            .fetch_one(&mut *transaction)\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to validate finalized header ledger: {e}\"))?;\n            anyhow::ensure!(\n                stored\n                    == (\n                        header.hash.clone(),\n                        header.parent_hash.clone(),","sourceCodeStart":4466,"sourceCodeEnd":4502,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L4466-L4502","documentation":"This error wraps a sqlx/PostgreSQL failure that occurs while inserting a subsequent verified finalized header (all headers after the first) into `execution_verified_finalized_header` during ledger extension. The insert uses `ON CONFLICT (chain_id, wallet_address, number) DO NOTHING`, so it fails only on genuine database-level errors, not on duplicate rows. The original driver error is preserved in the message via `{e}` and the operation runs inside a transaction, so any failure rolls back the whole bootstrap.","triggerScenarios":"Any PostgreSQL error during the INSERT for headers 2..n of `bootstrap.finalized_headers`: connection dropped mid-transaction, constraint violation other than the ignored unique conflict (e.g. NOT NULL or type errors from oversized/malformed hash or base_fee strings), lock timeout on the ledger rows, statement timeout, or the table/sequence being unavailable or migrated concurrently.","commonSituations":"Network blips between the adapter and PostgreSQL during long bootstrap transactions; schema drift after an upgrade (columns/added NOT NULL constraints); exceeding statement_timeout on very large finalized batches; database failover; connection pool exhaustion under concurrent bootstraps.","solutions":["Read the wrapped `{e}` cause to identify the concrete sqlx/Postgres error (connection, constraint, timeout) before changing anything.","Check database connectivity, pool limits, and statement/idle-in-transaction timeouts; retry the bootstrap on transient connection errors.","Compare the running schema against the expected `execution_verified_finalized_header` definition and apply pending migrations after version upgrades.","Reduce batch size or chunk the insert if statement_timeout is hit on large finalized ranges.","Inspect PostgreSQL server logs at failure time for locks, deadlocks, or failover events; ensure a single writer owns the bootstrap transaction."],"exampleFix":"// before: one huge transaction for the entire finalized range\nlet mut tx = pool.begin().await?;\nfor h in headers { insert_header(&mut tx, h).await?; }\n\n// after: bounded chunks with retry on transient errors\nfor chunk in headers.chunks(500) {\n    let mut tx = pool.begin().await?;\n    for h in chunk { insert_header(&mut tx, h).await?; }\n    tx.commit().await?; // smaller statements, shorter lock hold\n}","handlingStrategy":"retry","validationCode":"let reachable = sqlx::query(\"SELECT 1\").execute(&pool).await.is_ok();\nif !reachable { return Err(anyhow!(\"database unavailable before bootstrap\")); }\nlet schema_ok = sqlx::query(\n    \"SELECT 1 FROM information_schema.tables WHERE table_name = 'execution_verified_finalized_header'\",\n).fetch_optional(&pool).await?.is_some();\nif !schema_ok { return Err(anyhow!(\"ledger table missing; run migrations\")); }","typeGuard":null,"tryCatchPattern":"match insert_finalized_headers(&mut tx, &headers).await {\n    Err(e) if is_transient(&e) => backoff_retry(|| insert_finalized_headers(&mut tx, &headers), 3).await?,\n    Err(e) if e.to_string().contains(\"Failed to extend finalized header ledger\") => {\n        tracing::error!(cause = %e, \"ledger extension failed\");\n        return Err(e);\n    }\n    Ok(()) => {}\n}","preventionTips":["Run migrations as a deployment step before starting the adapter.","Size connection pools and statement timeouts for the largest expected finalized batch.","Chunk large inserts to keep transactions short.","Monitor PostgreSQL failover/lock events during bootstrap windows."],"tags":["database","postgres","sqlx","write-failure"],"backgroundTag":"database-write-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}