nautechsystems/nautilus_trader · critical · anyhow::Error

Execution payload marker exists without durable state

Error message

Execution payload marker exists without durable state

What it means

In `ensure_execution_payload_storage`, after confirming the schema-version marker exists, the durable `execution_payload_state` row must also exist. If the marker is present but the state row is missing, storage is in an inconsistent half-initialized state and this invariant-violation error is thrown instead of proceeding.

Source

Thrown at crates/adapters/blockchain/src/cache/database.rs:4625

    }

    /// Activates or resumes protected signed-transaction storage for this database.
    pub(crate) async fn ensure_execution_payload_storage(
        &self,
        keys: &PayloadKeySet,
    ) -> anyhow::Result<()> {
        let marker = self.execution_payload_marker().await?;
        match marker {
            None => self.activate_execution_payload_storage(keys).await?,
            Some(version) => anyhow::ensure!(
                version == EXECUTION_PAYLOAD_PROTOCOL_VERSION,
                "Execution payload protection version {version} is newer than supported version {EXECUTION_PAYLOAD_PROTOCOL_VERSION}"
            ),
        }

        loop {
            let state = self.execution_payload_state().await?.ok_or_else(|| {
                anyhow::anyhow!("Execution payload marker exists without durable state")
            })?;
            validate_execution_payload_state(&state, keys)?;
            match state.operation.as_str() {
                "migrate" => {
                    if self
                        .migrate_execution_payload_batch(keys, EXECUTION_PAYLOAD_BATCH_SIZE)
                        .await?
                    {
                        break;
                    }
                }
                "ready" => break,
                operation => anyhow::bail!(
                    "Execution payload storage is in {operation} maintenance; complete or roll back that operation before connecting"
                ),
            }
        }

View on GitHub (pinned to 18893faf8b)

Solutions

1. Re-create the durable state by running the activation path from a clean database (drop the payload tables and marker, then restart so activation re-runs)
2. Restore the missing execution_payload_state row from a backup taken while storage was healthy
3. Never delete state rows manually; use the library's own rollback/complete maintenance flow
4. Compare the current row set against a healthy database to identify what else is missing

Example fix

// before: manual removal breaks the invariant
DELETE FROM execution_payload_state WHERE component = 'signed_transactions';
// after: reset properly so activation re-runs
DELETE FROM execution_payload_state WHERE component = 'signed_transactions';
DELETE FROM execution_schema_version WHERE component = '<payload component>';

let marker: Option<i16> = sqlx::query_scalar(
    "SELECT version FROM execution_schema_version WHERE component = $1")
    .bind(EXECUTION_PAYLOAD_COMPONENT).fetch_optional(&pool).await?;
let state: Option<()> = sqlx::query_scalar(
    "SELECT 1 FROM execution_payload_state WHERE component = 'signed_transactions'")
    .fetch_optional(&pool).await?;
if marker.is_some() && state.is_none() {
    return Err(anyhow::anyhow!("marker present but state row missing; restore from backup or re-activate from clean state"));
}

fn storage_consistent(marker: Option<i16>, state_row: Option<impl Sized>) -> bool {
    marker.is_some() == state_row.is_some()
}

match db.ensure_execution_payload_storage(&keys).await {
    Err(e) if e.to_string().contains("marker exists without durable state") => {
        // halt node: storage is half-initialized; restore backup or reset both marker and state
        return Err(e.context("manual DB cleanup detected; re-provision payload storage"));
    }
    other => other?,
}

When it happens

Trigger: `execution_payload_state()` returns None while the marker row exists — e.g. someone deleted the `execution_payload_state` row for component 'signed_transactions', a partial manual cleanup removed state but not the version marker, or activation crashed in a way that committed the marker outside the state transaction.

Common situations: Manual DB surgery / cleanup scripts deleting rows; restoring a partial backup; an operator trying to 'reset' payload protection by deleting state rows.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Migrated terminal transaction hash was not found
• Execution payload marker exists without state
• Failed to load active execution intent: {e}
• Failed to start replacement transaction persistence: {e}
• Failed to lock active execution intent {intent_id}: {e}

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/d7a567e3868e7da1. Report an issue: GitHub.