nautechsystems/nautilus_trader · error · anyhow::Error

Failed to start replacement transaction persistence: {e}

Error message

Failed to start replacement transaction persistence: {e}

What it means

Wrapped sqlx error when pool.begin() fails at the start of add_execution_replacement_hash (database.rs:3791-3793), before any statement runs. Beginning a transaction first requires acquiring a pooled connection, so this almost always means the pool could not provide one: PoolTimedOut (acquire_timeout exceeded), PoolClosed, or the database is unreachable.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3791

    /// Attaches a canonical replacement which consumed the intent's signer nonce.
    ///
    /// The replacement bytes are unknown because standard JSON-RPC block responses expose
    /// decoded transaction fields, not the original signed envelope.
    ///
    /// # Errors
    ///
    /// Returns an error if the intent is not active, the hash conflicts, or persistence fails.
    pub async fn add_execution_replacement_hash(
        &self,
        intent_id: i64,
        chain_id: u32,
        transaction_hash: &str,
    ) -> anyhow::Result<ExecutionTransactionHashRow> {
        let chain_id_db = i32::try_from(chain_id)
            .with_context(|| format!("Chain ID {chain_id} exceeds PostgreSQL INTEGER"))?;
        let mut transaction = self.pool.begin().await.map_err(|e| {
            anyhow::anyhow!("Failed to start replacement transaction persistence: {e}")
        })?;
        let current_status = sqlx::query_scalar::<_, String>(
            "SELECT status FROM execution_intent WHERE id = $1 AND active FOR UPDATE",
        )
        .bind(intent_id)
        .fetch_optional(&mut *transaction)
        .await
        .map_err(|e| anyhow::anyhow!("Failed to lock active execution intent {intent_id}: {e}"))?
        .ok_or_else(|| anyhow::anyhow!("Active execution intent {intent_id} was not found"))?;
        anyhow::ensure!(
            execution_transition_allowed(&current_status, TransactionStatus::Replaced),
            "Invalid execution transition for intent {intent_id}: {current_status} -> replaced"
        );

        sqlx::query(
            "
            UPDATE execution_transaction_hash
            SET current = FALSE, status = 'replaced', updated_at = NOW()

View on GitHub (pinned to 2114cf6f76)

Solutions

1. Check the {e} cause: PoolTimedOut/PoolClosed point at pool sizing or lifecycle, Io/Connect at database reachability
2. Raise PoolOptions max_connections and/or acquire_timeout to cover the number of concurrent intent transactions
3. Verify Postgres is up and its own max_connections is not exhausted (compare against connection counts from other clients)
4. Retry with backoff - begin failing means no work was done, so the retry is trivially safe
5. During shutdown, stop accepting new replacement events before closing the pool

Example fix

// before: replacement observation dies with the pool hiccup
let row = db.add_execution_replacement_hash(intent_id, chain_id, &hash).await?;

// after: only retry connection-acquisition failures, nothing has executed yet
let row = loop {
    match db.add_execution_replacement_hash(intent_id, chain_id, &hash).await {
        Ok(row) => break row,
        Err(e) if is_transient_db_error(&e) => tokio::time::sleep(Duration::from_millis(200)).await,
        Err(e) => return Err(e),
    }
};

// pre-flight: confirm a connection is available before the event arrives
let _ = pool.acquire().await?; // fails fast with the same PoolTimedOut the begin would hit

fn is_pool_error(err: &anyhow::Error) -> bool {
    matches!(
        err.downcast_ref::<sqlx::Error>(),
        Some(sqlx::Error::PoolTimedOut) | Some(sqlx::Error::PoolClosed)
    )
}

match db.add_execution_replacement_hash(intent_id, chain_id, &hash).await {
    Ok(row) => Ok(row),
    Err(e) if is_pool_error(&e) => retry_with_backoff(e), // nothing executed yet: always safe
    Err(e) => Err(e),
}

When it happens

Trigger: Calling add_execution_replacement_hash while all pool connections are held by other intent transactions; acquire_timeout too small for the workload; Postgres down or at max_connections so no new connection can be established; pool closed during shutdown while a replacement event is still being processed.

Common situations: A burst of replacement/reorg events fan-out across watchers exhausting max_connections; Postgres max_connections reached because other services share the instance; graceful shutdown racing an in-flight replacement observation.

Related errors

• Failed to update execution hash {transaction_hash}: {e}
• Failed to update execution intent {intent_id}: {e}
• Failed to record execution transition: {e}
• Failed to retire replaced execution hash: {e}
• Failed to mark execution intent replaced: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@2114cf6f76 (2026-08-21). Data as JSON: /api/errors/d58a77f7e2efb4a4. Report an issue: GitHub.