What causes "Failed to start verified nonce assignment: {e}" in nautechsystems/nautilus_trader?

Check the wrapped source error (`{e}` in the message / the anyhow chain) for the root cause: connection refused, auth failed, or pool timeout.

nautechsystems/nautilus_trader · error · anyhow::Error

Failed to start verified nonce assignment: {e}

Error message

Failed to start verified nonce assignment: {e}

What it means

This error is raised while beginning a PostgreSQL transaction (via sqlx `pool.begin()`) to record a verified execution nonce assignment. It wraps the underlying connection/acquisition error with context so callers know the failure happened at the very start of the nonce-assignment flow. If the pool cannot establish or acquire a connection, the assignment is aborted before any row is read or written.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:5912

            "Verified nonce assignment requires decision evidence"
        );
        anyhow::ensure!(
            assignment.provider_ids.len() == 3 && assignment.operator_ids.len() == 3,
            "Verified nonce assignment requires exactly three provider and operator IDs"
        );
        anyhow::ensure!(
            assignment.failure_domain_ids.len() >= 3,
            "Verified nonce assignment requires the configured failure domains"
        );
        let chain_id = i32::try_from(assignment.chain_id)
            .context("Verification chain ID exceeds PostgreSQL INTEGER")?;
        let nonce =
            i64::try_from(assignment.nonce).context("Execution nonce exceeds PostgreSQL BIGINT")?;
        let mut transaction = self
            .pool
            .begin()
            .await
            .map_err(|e| anyhow::anyhow!("Failed to start verified nonce assignment: {e}"))?;
        let (manifest_version, manifest_digest, next_nonce, revision) =
            sqlx::query_as::<_, (String, String, i64, i64)>(
                "
                SELECT manifest_version, manifest_digest, next_canonical_nonce, revision
                FROM execution_verification_nonce
                WHERE chain_id = $1 AND wallet_address = $2
                FOR UPDATE
                ",
            )
            .bind(chain_id)
            .bind(assignment.wallet_address)
            .fetch_optional(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to lock canonical nonce ledger: {e}"))?
            .ok_or_else(|| anyhow::anyhow!("Canonical nonce ledger is not initialized"))?;
        anyhow::ensure!(
            manifest_version == assignment.manifest_version
                && manifest_digest == assignment.manifest_digest,

View on GitHub (pinned to 18893faf8b)

Solutions

Check the wrapped source error (`{e}` in the message / the anyhow chain) for the root cause: connection refused, auth failed, or pool timeout.
Verify the PostgreSQL host is reachable and healthy (`pg_isready`, docker ps, cloud console).
Increase pool limits or checkout timeout in the sqlx PoolOptions configuration to match concurrency.
If pool timeout, look for connection leaks (transactions not committed/rolled back elsewhere).
Confirm DSN/env vars (host, port, user, password, sslmode) are correct for the environment.

Example fix

// before
 DatabaseConfig::new(dsn).max_connections(5)
// after
 DatabaseConfig::new(dsn).max_connections(20).acquire_timeout(Duration::from_secs(10))

Defensive patterns

Strategy: try-catch

Validate before calling

let healthy = sqlx::query("SELECT 1").fetch_one(&pool).await.is_ok();
if !healthy { return Err(anyhow!("database pool unavailable before nonce assignment")); }

Try / catch

match db.assign_verified_nonce(&assignment).await {
    Ok(()) => info!("nonce assigned"),
    Err(e) if e.to_string().contains("Failed to start verified nonce assignment") => {
        warn!("DB transaction could not start; retrying after health check");
        wait_for_db(&pool).await?;
        retry(assignment)
    }
    Err(e) => return Err(e),
}

Prevention

Run a startup health check (SELECT 1) before processing work that needs the database.
Size the pool (max_connections, acquire_timeout) for your actual concurrency.
Monitor pool checkout saturation and DB connectivity with metrics/alerts.
Use bounded retry with backoff around DB-bound operations.
Use TCP keepalives on Postgres connections to survive idle network timeouts.

When it happens

Trigger: Calling the verified-nonce-assignment persistence path when `self.pool.begin()` fails: database unreachable, connection pool exhausted (all checkouts in use / timeout), TLS or auth failure, or the database was restarted between pool creation and this call.

Common situations: Postgres container down or restarted during a trading session; pool max_connections too low for concurrent strategies; network partition to the DB host; wrong DSN credentials in environment config; firewall/idle timeouts killing pooled connections.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/1666a68ff7f36034. Report an issue: GitHub.