{"record":{"id":"b4076d3f2d196ab0","repo":"nautechsystems/nautilus_trader","slug":"failed-to-start-execution-verification-migration","errorCode":null,"errorMessage":"Failed to start execution verification migration: {e}","messagePattern":"Failed to start execution verification migration: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":3897,"sourceCode":"                && bootstrap.failure_domain_ids.len() >= 3\n                && !bootstrap.decisions.is_empty(),\n            \"Connect verification evidence is incomplete\"\n        );\n        let chain_id = i32::try_from(bootstrap.chain_id)\n            .context(\"Verification chain ID exceeds PostgreSQL INTEGER\")?;\n        let checkpoint_number = i64::try_from(bootstrap.checkpoint_number)\n            .context(\"Verification checkpoint exceeds PostgreSQL BIGINT\")?;\n        let checkpoint_timestamp = i64::try_from(bootstrap.checkpoint_timestamp)\n            .context(\"Verification checkpoint timestamp exceeds PostgreSQL BIGINT\")?;\n        let next_canonical_nonce = i64::try_from(bootstrap.next_canonical_nonce)\n            .context(\"Canonical nonce exceeds PostgreSQL BIGINT\")?;\n        let observed_canonical_nonce = i64::try_from(bootstrap.observed_canonical_nonce)\n            .context(\"Observed canonical nonce exceeds PostgreSQL BIGINT\")?;\n        let base_fee = bootstrap\n            .checkpoint_base_fee_per_gas\n            .map(|value| value.to_string());\n        let mut transaction = self.pool.begin().await.map_err(|e| {\n            anyhow::anyhow!(\"Failed to start execution verification migration: {e}\")\n        })?;\n\n        for statement in [\n            \"\n            CREATE TABLE IF NOT EXISTS execution_verification_nonce (\n                chain_id INTEGER NOT NULL REFERENCES chain(chain_id) ON DELETE RESTRICT,\n                wallet_address TEXT NOT NULL,\n                manifest_version TEXT NOT NULL CHECK (manifest_version <> ''),\n                manifest_digest TEXT NOT NULL CHECK (manifest_digest <> ''),\n                next_canonical_nonce BIGINT NOT NULL CHECK (next_canonical_nonce >= 0),\n                revision BIGINT NOT NULL DEFAULT 0 CHECK (revision >= 0),\n                updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),\n                PRIMARY KEY (chain_id, wallet_address)\n            )\n            \",\n            \"\n            CREATE TABLE IF NOT EXISTS execution_verified_finalized_header (\n                chain_id INTEGER NOT NULL,","sourceCodeStart":3879,"sourceCodeEnd":3915,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/cache/database.rs#L3879-L3915","documentation":"This error wraps a failure from `self.pool.begin()` while starting the PostgreSQL transaction that performs the execution verification migration (`ensure_execution_verification_schema`). sqlx could not open a transaction on the connection pool, and the underlying sqlx error is embedded in the message as `{e}`. It is a pre-statement failure: none of the schema DDL has run yet.","triggerScenarios":"Calling `ensure_execution_verification_schema` when the connection pool cannot begin a transaction — pool exhausted (all connections busy), database unreachable, TLS/auth rejected, or the connection dropped mid-handshake.","commonSituations":"PostgreSQL is down or restarting during application startup; wrong host/port/password in `DATABASE_URL`; connection pool `max_connections` too small for concurrent startup tasks; firewall or idle timeout severing pooled connections; certificate/TLS misconfiguration.","solutions":["Read the wrapped sqlx error in the message: if it says connection refused/timeout, verify PostgreSQL is running and `DATABASE_URL` (host, port, user, password, dbname) is correct.","Increase the sqlx pool limits (`max_connections`, `acquire_timeout`) or reduce concurrent startup work that starves the pool.","Check network reachability and TLS configuration between the adapter and the database (firewall rules, security groups, CA certificates).","If errors are intermittent after idle periods, enable pool idle timeouts/recycling so stale connections are not reused."],"exampleFix":"// before\nlet pool = PgPool::connect(&database_url).await?; // default pool, small acquire timeout\n\n// after\nlet pool = PgPoolOptions::new()\n    .max_connections(10)\n    .acquire_timeout(std::time::Duration::from_secs(30))\n    .idle_timeout(std::time::Duration::from_secs(300))\n    .connect(&database_url)\n    .await?;","handlingStrategy":"retry","validationCode":"async fn assert_pool_ready(pool: &sqlx::PgPool) -> Result<(), sqlx::Error> {\n    let mut tx = pool.begin().await?; // fail early with the raw driver error\n    tx.rollback().await?;\n    Ok(())\n}","typeGuard":null,"tryCatchPattern":"match database.ensure_execution_verification_schema(&bootstrap).await {\n    Err(e) if e.to_string().contains(\"Failed to start execution verification migration\") => {\n        // inspect wrapped sqlx error; backoff and retry for transient connect/pool-acquire failures\n        tokio::time::sleep(Duration::from_secs(5)).await;\n        database.ensure_execution_verification_schema(&bootstrap).await?;\n    }\n    Err(e) => return Err(e),\n    Ok(()) => {}\n}","preventionTips":["Validate DATABASE_URL connectivity with a trivial query before starting the migration path.","Size the pool (max_connections, acquire_timeout) for concurrent startup workloads.","Monitor PostgreSQL availability and enable connection recycling to avoid stale pooled connections.","Retry only transient errors (connection reset, acquire timeout); surface auth/config errors immediately."],"tags":["database","postgresql","connection","transaction"],"backgroundTag":"database-query-failed","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}