nautechsystems/nautilus_trader · error · anyhow::Error

Connect verification evidence is incomplete

Error message

Connect verification evidence is incomplete

What it means

`ensure_execution_verification_schema` requires the bootstrap's connect-verification evidence to be complete: exactly 3 provider IDs, exactly 3 operator IDs, at least 3 failure-domain IDs, and at least one recorded decision. This `anyhow::ensure!` fires when any of those conditions is unmet, meaning the quorum evidence backing the verification ledger is missing or insufficient. The schema's own CHECK constraints (`cardinality(provider_ids) = 3`, etc.) mirror this requirement, so incomplete evidence would be rejected downstream anyway.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:3876

    ///
    /// Returns an error if the schema, migration snapshot, historical reconstruction, or retained
    /// evidence is inconsistent, or persistence fails.
    pub(crate) async fn ensure_execution_verification_schema(
        &self,
        bootstrap: &ExecutionVerificationBootstrap<'_>,
    ) -> anyhow::Result<()> {
        let first_header = bootstrap
            .finalized_headers
            .first()
            .ok_or_else(|| anyhow::anyhow!("Verified finalized header ledger is empty"))?;
        anyhow::ensure!(
            bootstrap.finalized_headers.windows(2).all(|headers| {
                headers[1].number == headers[0].number.saturating_add(1)
                    && headers[1].parent_hash == headers[0].hash
            }),
            "Verified finalized headers are not one continuous parent-linked chain"
        );
        anyhow::ensure!(
            bootstrap.provider_ids.len() == 3
                && bootstrap.operator_ids.len() == 3
                && bootstrap.failure_domain_ids.len() >= 3
                && !bootstrap.decisions.is_empty(),
            "Connect verification evidence is incomplete"
        );
        let chain_id = i32::try_from(bootstrap.chain_id)
            .context("Verification chain ID exceeds PostgreSQL INTEGER")?;
        let checkpoint_number = i64::try_from(bootstrap.checkpoint_number)
            .context("Verification checkpoint exceeds PostgreSQL BIGINT")?;
        let checkpoint_timestamp = i64::try_from(bootstrap.checkpoint_timestamp)
            .context("Verification checkpoint timestamp exceeds PostgreSQL BIGINT")?;
        let next_canonical_nonce = i64::try_from(bootstrap.next_canonical_nonce)
            .context("Canonical nonce exceeds PostgreSQL BIGINT")?;
        let observed_canonical_nonce = i64::try_from(bootstrap.observed_canonical_nonce)
            .context("Observed canonical nonce exceeds PostgreSQL BIGINT")?;
        let base_fee = bootstrap
            .checkpoint_base_fee_per_gas

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Ensure exactly three providers and three operators are configured and reachable, with at least three distinct failure domains, before building the bootstrap.
  2. Complete the connect-verification flow so at least one decision is recorded before invoking `ensure_execution_verification_schema`.
  3. Check provider connectivity — if fewer than three providers respond, the quorum evidence will be short and this error fires.
  4. Review how provider/operator/failure-domain IDs are aggregated into the bootstrap; a filtering bug can silently drop entries below the required counts.

Example fix

// before
let bootstrap = ExecutionVerificationBootstrap {
    provider_ids: &responded_providers,   // maybe only 2 responded
    operator_ids: &operators,
    failure_domain_ids: &domains,
    decisions: &decisions,
    ..b
};

// after
anyhow::ensure!(
    responded_providers.len() == 3
        && operators.len() == 3
        && domains.len() >= 3
        && !decisions.is_empty(),
    "connect verification evidence incomplete before migration"
);
let bootstrap = ExecutionVerificationBootstrap {
    provider_ids: &responded_providers,
    operator_ids: &operators,
    failure_domain_ids: &domains,
    decisions: &decisions,
    ..b
};
Defensive patterns

Strategy: validation

Validate before calling

fn validate_connect_evidence(b: &ExecutionVerificationBootstrap<'_>) -> Result<(), String> {
    match (
        b.provider_ids.len() == 3,
        b.operator_ids.len() == 3,
        b.failure_domain_ids.len() >= 3,
        !b.decisions.is_empty(),
    ) {
        (true, true, true, true) => Ok(()),
        (p, o, f, d) => Err(format!(
            "incomplete evidence: providers={} (need 3), operators={} (need 3), failure_domains>=3={}, decisions_nonempty={}",
            b.provider_ids.len(), b.operator_ids.len(), f, d
        )),
    }
}

Type guard

fn has_complete_connect_evidence(b: &ExecutionVerificationBootstrap<'_>) -> bool {
    b.provider_ids.len() == 3
        && b.operator_ids.len() == 3
        && b.failure_domain_ids.len() >= 3
        && !b.decisions.is_empty()
}

Try / catch

match database.ensure_execution_verification_schema(&bootstrap).await {
    Err(e) if e.to_string().contains("Connect verification evidence is incomplete") => {
        // check provider/operator configuration and connect-verification completion, rebuild evidence, retry
    }
    Err(e) => return Err(e),
    Ok(()) => {}
}

Prevention

When it happens

Trigger: Constructing `ExecutionVerificationBootstrap` where `provider_ids.len() != 3`, `operator_ids.len() != 3`, `failure_domain_ids.len() < 3`, or `decisions.is_empty()` — e.g. only one or two providers responded during connect verification, or no decisions were recorded before migration.

Common situations: A misconfigured provider list (fewer than 3 RPC endpoints); providers being unreachable during connect verification so quorum never completes; an operator/failure-domain topology not yet fully registered; running migration before any connect-verification decisions have been persisted.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/a34632ba0f93456f. Report an issue: GitHub.