{"record":{"id":"d58a77f7e2efb4a4","repo":"nautechsystems/nautilus_trader","slug":"failed-to-start-replacement-transaction-persistenc","errorCode":null,"errorMessage":"Failed to start replacement transaction persistence: {e}","messagePattern":"Failed to start replacement transaction persistence: (.+?)","errorType":"exception","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/cache/database.rs","lineNumber":3791,"sourceCode":"\n    /// Attaches a canonical replacement which consumed the intent's signer nonce.\n    ///\n    /// The replacement bytes are unknown because standard JSON-RPC block responses expose\n    /// decoded transaction fields, not the original signed envelope.\n    ///\n    /// # Errors\n    ///\n    /// Returns an error if the intent is not active, the hash conflicts, or persistence fails.\n    pub async fn add_execution_replacement_hash(\n        &self,\n        intent_id: i64,\n        chain_id: u32,\n        transaction_hash: &str,\n    ) -> anyhow::Result<ExecutionTransactionHashRow> {\n        let chain_id_db = i32::try_from(chain_id)\n            .with_context(|| format!(\"Chain ID {chain_id} exceeds PostgreSQL INTEGER\"))?;\n        let mut transaction = self.pool.begin().await.map_err(|e| {\n            anyhow::anyhow!(\"Failed to start replacement transaction persistence: {e}\")\n        })?;\n        let current_status = sqlx::query_scalar::<_, String>(\n            \"SELECT status FROM execution_intent WHERE id = $1 AND active FOR UPDATE\",\n        )\n        .bind(intent_id)\n        .fetch_optional(&mut *transaction)\n        .await\n        .map_err(|e| anyhow::anyhow!(\"Failed to lock active execution intent {intent_id}: {e}\"))?\n        .ok_or_else(|| anyhow::anyhow!(\"Active execution intent {intent_id} was not found\"))?;\n        anyhow::ensure!(\n            execution_transition_allowed(&current_status, TransactionStatus::Replaced),\n            \"Invalid execution transition for intent {intent_id}: {current_status} -> replaced\"\n        );\n\n        sqlx::query(\n            \"\n            UPDATE execution_transaction_hash\n            SET current = FALSE, status = 'replaced', updated_at = NOW()","sourceCodeStart":3773,"sourceCodeEnd":3809,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/2114cf6f761429e0adb5ca9596fcd7b895b16011/crates/adapters/blockchain/src/cache/database.rs#L3773-L3809","documentation":"Wrapped sqlx error when pool.begin() fails at the start of add_execution_replacement_hash (database.rs:3791-3793), before any statement runs. Beginning a transaction first requires acquiring a pooled connection, so this almost always means the pool could not provide one: PoolTimedOut (acquire_timeout exceeded), PoolClosed, or the database is unreachable.","triggerScenarios":"Calling add_execution_replacement_hash while all pool connections are held by other intent transactions; acquire_timeout too small for the workload; Postgres down or at max_connections so no new connection can be established; pool closed during shutdown while a replacement event is still being processed.","commonSituations":"A burst of replacement/reorg events fan-out across watchers exhausting max_connections; Postgres max_connections reached because other services share the instance; graceful shutdown racing an in-flight replacement observation.","solutions":["Check the {e} cause: PoolTimedOut/PoolClosed point at pool sizing or lifecycle, Io/Connect at database reachability","Raise PoolOptions max_connections and/or acquire_timeout to cover the number of concurrent intent transactions","Verify Postgres is up and its own max_connections is not exhausted (compare against connection counts from other clients)","Retry with backoff - begin failing means no work was done, so the retry is trivially safe","During shutdown, stop accepting new replacement events before closing the pool"],"exampleFix":"// before: replacement observation dies with the pool hiccup\nlet row = db.add_execution_replacement_hash(intent_id, chain_id, &hash).await?;\n\n// after: only retry connection-acquisition failures, nothing has executed yet\nlet row = loop {\n    match db.add_execution_replacement_hash(intent_id, chain_id, &hash).await {\n        Ok(row) => break row,\n        Err(e) if is_transient_db_error(&e) => tokio::time::sleep(Duration::from_millis(200)).await,\n        Err(e) => return Err(e),\n    }\n};","handlingStrategy":"retry","validationCode":"// pre-flight: confirm a connection is available before the event arrives\nlet _ = pool.acquire().await?; // fails fast with the same PoolTimedOut the begin would hit","typeGuard":"fn is_pool_error(err: &anyhow::Error) -> bool {\n    matches!(\n        err.downcast_ref::<sqlx::Error>(),\n        Some(sqlx::Error::PoolTimedOut) | Some(sqlx::Error::PoolClosed)\n    )\n}","tryCatchPattern":"match db.add_execution_replacement_hash(intent_id, chain_id, &hash).await {\n    Ok(row) => Ok(row),\n    Err(e) if is_pool_error(&e) => retry_with_backoff(e), // nothing executed yet: always safe\n    Err(e) => Err(e),\n}","preventionTips":["Size max_connections for peak concurrent intent transactions, not average load","Set acquire_timeout deliberately and monitor acquisition latency","Stop enqueueing replacement events before closing the pool on shutdown","Watch Postgres-side connection counts if the instance is shared"],"tags":["rust","sqlx","postgres","blockchain","connection-pool","transaction"],"backgroundTag":"db-transaction-begin-failed","analyzedSha":"2114cf6f761429e0adb5ca9596fcd7b895b16011","analyzedAt":"2026-08-21T11:28:30.864Z","schemaVersion":2},"datasetVersion":"2026-08-22T04:17:13.399Z"}