{"record":{"id":"a8912bee5da3312e","repo":"nautechsystems/nautilus_trader","slug":"failed-to-start-execution-schema-migration-e","errorCode":null,"errorMessage":"Failed to start execution schema migration: {e}","messagePattern":"Failed to start execution schema migration: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":3049,"sourceCode":"        Ok(())\n    }\n\n    /// Installs execution schema version 2 without changing existing transaction rows.\n    ///\n    /// The migration locks the legacy transaction table, refuses unresolved version 1 rows,\n    /// installs the versioned intent and hash-history tables, and fences older writers before\n    /// releasing the lock. This prevents a mixed-version process from bypassing the new signer\n    /// ownership constraints.\n    ///\n    /// # Errors\n    ///\n    /// Returns an error if the database operation fails.\n    pub async fn ensure_execution_transaction_schema(&self) -> anyhow::Result<()> {\n        let mut transaction = self\n            .pool\n            .begin()\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to start execution schema migration: {e}\"))?;\n\n        sqlx::query(\"LOCK TABLE execution_transaction IN ACCESS EXCLUSIVE MODE\")\n            .execute(&mut *transaction)\n            .await\n            .map_err(|e| anyhow::anyhow!(\"Failed to lock legacy execution transactions: {e}\"))?;\n\n        for statement in [\n            \"\n            ALTER TABLE execution_transaction\n            ADD COLUMN IF NOT EXISTS client_order_id TEXT\n            \",\n            \"\n            ALTER TABLE execution_transaction\n            ADD COLUMN IF NOT EXISTS wallet_address TEXT\n            \",\n            \"\n            ALTER TABLE execution_transaction\n            ALTER COLUMN wallet_address DROP NOT NULL","sourceCodeStart":3031,"sourceCodeEnd":3067,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/2114cf6f761429e0adb5ca9596fcd7b895b16011/crates/adapters/blockchain/src/cache/database.rs#L3031-L3067","documentation":"Thrown when BlockchainCacheDatabase::ensure_execution_transaction_schema fails at pool.begin(), before any migration SQL runs. The wrapped sqlx error means a connection could not be acquired: Postgres unreachable, pool exhausted, or the connection was dropped. The schema is untouched when this fires, so the operation is safely retryable.","triggerScenarios":"Calling ensure_execution_transaction_schema at startup while Postgres is down or DATABASE_URL/DSN points at the wrong host; a sqlx pool sized above Postgres max_connections; every pooled connection checked out by trading workers so begin() times out on acquire.","commonSituations":"Service starts before the Postgres container passes readiness checks; misconfigured credentials or port; a connection leak elsewhere exhausts the pool; failover happens mid-startup.","solutions":["Verify Postgres is reachable with the exact DSN used by the node (psql or SELECT 1) and fix host/port/credentials","Run the migration once at startup, before workers take connections, instead of concurrently with trading activity","Size the sqlx pool (max_connections/acquire_timeout) within Postgres max_connections","Add startup retry with backoff so the node waits for database readiness instead of aborting"],"exampleFix":null,"handlingStrategy":"retry","validationCode":"// Gate startup on database readiness before invoking the migration\nlet ready = sqlx::query(\"SELECT 1\").execute(&pool).await.is_ok();\nif !ready {\n    // wait / re-probe with backoff instead of calling ensure_execution_transaction_schema\n}","typeGuard":null,"tryCatchPattern":"fn is_transient(e: &sqlx::Error) -> bool {\n    matches!(\n        e,\n        sqlx::Error::Io(_)\n            | sqlx::Error::PoolTimedOut\n            | sqlx::Error::ConnectionClosed(_)\n            | sqlx::Error::Database(_)\n    ) && !matches!(e, sqlx::Error::Database(_))\n}\nmatch db.ensure_execution_transaction_schema().await {\n    Ok(()) => {}\n    Err(e) if e.downcast_ref::<sqlx::Error>().is_some_and(is_transient) => {\n        // backoff and retry the idempotent migration\n    }\n    Err(e) => return Err(e),\n}","preventionTips":["Run ensure_execution_transaction_schema once at startup, before any worker acquires pool connections","Keep the sqlx pool max_connections below Postgres max_connections with headroom for admin sessions","Add a database readiness gate (SELECT 1 loop) to deployment scripts and container startup orders","Alert on pool acquire_timeout metrics so exhaustion is visible before begin() calls fail"],"tags":["rust","sqlx","postgresql","connection-pool","migration","startup"],"backgroundTag":"database-connection-failed","analyzedSha":"2114cf6f761429e0adb5ca9596fcd7b895b16011","analyzedAt":"2026-08-21T11:28:30.864Z","schemaVersion":2},"datasetVersion":"2026-08-22T04:17:13.399Z"}