nautechsystems/nautilus_trader · error
Failed to start recoverable execution transition
Error message
Failed to start recoverable execution transition: {e} What it means
`mark_execution_intent_recoverable` begins by opening a PostgreSQL transaction via `self.pool.begin()`. If acquiring a connection/starting the transaction fails (pool exhausted, database down, TLS/network error), the underlying sqlx error is wrapped into this anyhow error and returned before any intent state is touched.
Solutions
- Check database connectivity and credentials, then retry the operation — it is safe to retry since nothing was modified.
- Increase the sqlx pool size or add acquire-timeout headroom to handle pool exhaustion.
- Inspect the wrapped inner error (`{e}`) to distinguish network vs auth vs server errors.
- Add connection health checks / keepalives so stale connections are replaced before `begin()`.
Example fix
// before: no retry around transient pool errors
for id in intents { db.mark_execution_intent_recoverable(id).await?; }
// after: bounded retry on transient failures
for id in intents {
for _ in 0..3 {
match db.mark_execution_intent_recoverable(id).await {
Ok(()) => break,
Err(e) if is_transient_db_error(&e) => tokio::time::sleep(Duration::from_millis(200)).await,
Err(e) => return Err(e),
}
}
} Defensive patterns
Strategy: retry
Validate before calling
sqlx::query("SELECT 1").execute(&pool).await.map_err(|e| anyhow!("database unreachable: {e}"))?; Try / catch
match db.mark_execution_intent_recoverable(id).await {
Err(e) if is_transient_db_error(&e) => { backoff_and_retry(id, 3).await?; }
Err(e) => return Err(e),
Ok(()) => {},
} Prevention
- Size the connection pool for peak concurrency and set a sane acquire timeout.
- Monitor pool wait metrics to catch saturation before it surfaces as begin() failures.
- Validate DSN/credentials at startup so auth failures do not appear mid-lifecycle.
- Make recoverable-transition calls idempotent so they are safe to retry after transient DB errors.
When it happens
Trigger: Any call to `mark_execution_intent_recoverable(intent_id)` while the database connection pool is exhausted, the database is unreachable, credentials are invalid, or the server rejects `BEGIN`.
Common situations: Connection pool saturation under load; PostgreSQL restart or failover; network partition between the adapter and the database; wrong DSN or expired credentials in the environment.
Understand the failure class
Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.
Related errors
- Error connecting to Postgres
- Failed to lock recoverable execution intent
- Failed to mark execution intent recoverable
- Failed to set synchronous_commit ON
- Failed to start execution status transition
AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08).
Data as JSON: /api/errors/6f128a0caefcf50e.
Report an issue: GitHub.
Appendix: source
Thrown at crates/adapters/blockchain/src/cache/database.rs:6512
.context("failed to record the verified replacement transition")?;
}
}
transaction
.commit()
.await
.context("failed to commit verified replacement scan")?;
Ok(())
}
/// Releases an intent when no broadcast attempt can have occurred.
///
/// # Errors
///
/// Returns an error if the intent advanced to broadcast or persistence fails.
pub async fn mark_execution_intent_recoverable(&self, intent_id: i64) -> anyhow::Result<()> {
let mut transaction = self.pool.begin().await.map_err(|e| {
anyhow::anyhow!("Failed to start recoverable execution transition: {e}")
})?;
let current_status = sqlx::query_scalar::<_, String>(
"SELECT status FROM execution_intent WHERE id = $1 FOR UPDATE",
)
.bind(intent_id)
.fetch_optional(&mut *transaction)
.await
.map_err(|e| anyhow::anyhow!("Failed to lock recoverable execution intent: {e}"))?
.ok_or_else(|| anyhow::anyhow!("Execution intent {intent_id} was not found"))?;
anyhow::ensure!(
current_status == "prepared",
"Execution intent {intent_id} is {current_status}, not recoverable before signing"
);
let result = sqlx::query(
"
UPDATE execution_intent
SET status = 'recoverable', active = FALSE, updated_at = NOW()
WHERE id = $1 AND status = 'prepared'View on GitHub (pinned to 18893faf8b)