What causes "Failed to start verified action evidence: {e}" in nautechsystems/nautilus_trader?

Verify database connectivity and credentials, then retry the operation.

Failed to start verified action evidence: {e}

Error message

Failed to start verified action evidence: {e}

What it means

Thrown when opening the PostgreSQL transaction for recording verified-action evidence fails at the pool.begin() step. The database could not start a transaction, so no ledger read or writes occur and the whole operation aborts with this anyhow error.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:6066

        anyhow::ensure!(
            !batch.decision_class.trim().is_empty() && !batch.decisions.is_empty(),
            "Verified action requires a decision class and evidence"
        );
        anyhow::ensure!(
            batch.provider_ids.len() == 3
                && batch.operator_ids.len() == 3
                && batch.failure_domain_ids.len() >= 3,
            "Verified action requires the configured provider identities"
        );
        let chain_id = i32::try_from(batch.chain_id)
            .context("Verification chain ID exceeds PostgreSQL INTEGER")?;
        let nonce =
            i64::try_from(batch.nonce).context("Execution nonce exceeds PostgreSQL BIGINT")?;
        let mut transaction = self
            .pool
            .begin()
            .await
            .map_err(|e| anyhow::anyhow!("Failed to start verified action evidence: {e}"))?;
        let (manifest_version, manifest_digest, revision) =
            sqlx::query_as::<_, (String, String, i64)>(
                "
                SELECT manifest_version, manifest_digest, revision
                FROM execution_verification_nonce
                WHERE chain_id = $1 AND wallet_address = $2
                FOR SHARE
                ",
            )
            .bind(chain_id)
            .bind(batch.wallet_address)
            .fetch_optional(&mut *transaction)
            .await
            .map_err(|e| anyhow::anyhow!("Failed to read verified action nonce ledger: {e}"))?
            .ok_or_else(|| anyhow::anyhow!("Canonical nonce ledger is not initialized"))?;
        anyhow::ensure!(
            manifest_version == batch.manifest_version && manifest_digest == batch.manifest_digest,
            "Verified action manifest identity changed"

View on GitHub (pinned to 18893faf8b)

Solutions

Verify database connectivity and credentials, then retry the operation.
Increase the sqlx pool size or add acquire_timeout headroom if the pool is saturated.
Check Postgres logs and health (pg_isready) for restarts or failures at the error timestamp.
Retry with backoff; beginning a transaction is safe to retry since no work was done.

Example fix

// before
record_execution_verification_batch(&db, &batch).await?;
// after: ensure the pool is healthy first
sqlx::query("SELECT 1").execute(&db).await.context("database unavailable")?;
record_execution_verification_batch(&db, &batch).await?;

Defensive patterns

Strategy: retry

Validate before calling

// Rust: health-check the pool before the operation
sqlx::query("SELECT 1").execute(&db).await
    .context("database unavailable before verified action")?;

Try / catch

// Rust
match record_batch().await {
    Err(e) if e.to_string().contains("Failed to start verified action evidence") => {
        wait_for_db_ready(&db).await?;
        record_batch().await.map_err(Into::into)
    }
    res => res.map_err(Into::into),
}

Prevention

Monitor pool utilization and raise max_size under load.
Add connection health checks and reasonable acquire_timeout.
Alert on database restarts/failovers.

When it happens

Trigger: self.pool.begin() returns Err — pool exhausted, database unreachable, connection broken, or the database is starting up/shutting down.

Common situations: Postgres restart or failover; connection pool max_size exhausted under load; network partition to the database; wrong DSN/credentials after a config change so all connections fail.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/a6f290b6d5295e84. Report an issue: GitHub.