nautechsystems/nautilus_trader · error · anyhow::Error

Profiler receipt position does not match its ingestion water

Error message

Profiler receipt position does not match its ingestion watermark

What it means

This is a consistency check in the execution client's profiler watermark verification. After fetching a transaction receipt, the library verifies that the receipt's block number, block hash, and transaction index exactly match the ingestion watermark position used to locate it. If the receipt was mined/reorged into a different position than expected, this error is thrown to prevent building the profiler watermark on stale or inconsistent data.

Source

Thrown at crates/adapters/blockchain/src/execution/client.rs:4202

    let transaction_hash = B256::from_str(&position.transaction_hash).with_context(|| {
        format!(
            "Invalid profiler transaction hash {}",
            position.transaction_hash
        )
    })?;
    let receipt = verified_value(
        verification.verify_receipt(&transaction_hash).await,
        "profiler watermark receipt",
    )?;
    anyhow::ensure!(
        receipt.status,
        "Profiler transaction did not execute successfully"
    );
    anyhow::ensure!(
        receipt.transaction_hash == transaction_hash,
        "Profiler receipt transaction hash does not match its ingestion watermark"
    );
    anyhow::ensure!(
        receipt.block_number == position.number
            && receipt.block_hash == expected_block_hash
            && receipt.transaction_index == u64::from(position.transaction_index),
        "Profiler receipt position does not match its ingestion watermark"
    );
    let matching_logs = receipt
        .logs
        .iter()
        .filter(|log| rpc_log::extract_log_index(log).ok() == Some(position.log_index))
        .collect::<Vec<_>>();
    anyhow::ensure!(
        matching_logs.len() == 1,
        "Profiler receipt contains {} logs at global index {}; expected exactly one",
        matching_logs.len(),
        position.log_index
    );
    let log = matching_logs[0];
    let log_transaction_hash = B256::from_str(&rpc_log::extract_transaction_hash(log)?)

View on GitHub (pinned to 18893faf8b)

Solutions

  1. Re-run the profiler watermark validation after the reorg settles; confirm the block is stable (wait confirmations).
  2. Point the execution RPC endpoint to a node consistent with the data feed (same provider/region, no mixed fallback nodes).
  3. Verify expected_block_hash and position come from the same latest-block query, not cached/stale state.
  4. Pin a single RPC endpoint instead of a round-robin load balancer to avoid cross-node inconsistency.

Example fix

// before: single-shot fetch, no reorg tolerance
let receipt = rpc.get_transaction_receipt(tx_hash).await?;
validate_receipt_position(&receipt, &position, expected_block_hash)?;

// after: retry on reorg with fresh position
for _ in 0..3 {
    let (position, expected_block_hash) = fetch_latest_position(&rpc).await?;
    let receipt = rpc.get_transaction_receipt(tx_hash).await?;
    match validate_receipt_position(&receipt, &position, expected_block_hash) {
        Ok(()) => break,
        Err(_) => tokio::time::sleep(Duration::from_secs(2)).await,
    }
}
Defensive patterns

Strategy: retry

Validate before calling

let latest = rpc.get_block_number().await?;
assert!(position.number <= latest, "watermark position ahead of chain");
assert_eq!(receipt.block_number, position.number);

Type guard

fn receipt_matches_position(receipt: &Receipt, position: &Position) -> bool {
    receipt.block_number == position.number
        && receipt.transaction_index as u64 == position.transaction_index
}

Try / catch

match client.fetch_profiler_receipt(tx_hash).await {
    Ok(r) => r,
    Err(e) if e.to_string().contains("receipt position does not match") => retry_after_confirmations(e),
    Err(e) => return Err(e),
}

Prevention

When it happens

Trigger: Calling the profiler watermark validation path (validate receipt against position) when the receipt's block_number != position.number, receipt.block_hash != expected_block_hash, or receipt.transaction_index != position.transaction_index. Typically triggered by a chain reorg between watermark ingestion and receipt fetch, or querying a different/inconsistent RPC endpoint.

Common situations: Reorged blocks on a testnet/local fork; load-balanced RPC endpoints backed by out-of-sync nodes; an execution RPC endpoint lagging behind the data feed; running the profiler against a snapshot that has since been reorganized.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/c739b7f4b4bc84b0. Report an issue: GitHub.