nautechsystems/nautilus_trader · error · anyhow::Error

Profiler log position does not match its ingestion watermark

Error message

Profiler log position does not match its ingestion watermark

What it means

After locating the watermark log, the library verifies the log's removed flag is false and its block number, transaction index, transaction hash, and block hash match the ingestion watermark. This error means the log's on-chain position disagrees with the recorded watermark position — usually the result of a reorg or cross-node inconsistency.

Source

Thrown at crates/adapters/blockchain/src/execution/client.rs:4226

    let matching_logs = receipt
        .logs
        .iter()
        .filter(|log| rpc_log::extract_log_index(log).ok() == Some(position.log_index))
        .collect::<Vec<_>>();
    anyhow::ensure!(
        matching_logs.len() == 1,
        "Profiler receipt contains {} logs at global index {}; expected exactly one",
        matching_logs.len(),
        position.log_index
    );
    let log = matching_logs[0];
    let log_transaction_hash = B256::from_str(&rpc_log::extract_transaction_hash(log)?)
        .with_context(|| "Invalid profiler log transaction hash")?;
    let log_block_hash = log
        .block_hash
        .as_deref()
        .ok_or_else(|| anyhow::anyhow!("Profiler log has no block hash"))?;
    anyhow::ensure!(
        !log.removed
            && log_transaction_hash == transaction_hash
            && rpc_log::extract_block_number(log)? == position.number
            && rpc_log::extract_transaction_index(log)? == position.transaction_index
            && B256::from_str(log_block_hash)? == expected_block_hash,
        "Profiler log position does not match its ingestion watermark"
    );
    anyhow::ensure!(
        rpc_log::extract_address(log)? == pool_address,
        "Profiler watermark log did not come from expected pool {pool_address}"
    );
    let signature = log
        .topics
        .first()
        .ok_or_else(|| anyhow::anyhow!("Profiler watermark log has no event signature"))?;
    let supported =
        profiler_event_signatures(pool).any(|expected| expected.eq_ignore_ascii_case(signature));
    anyhow::ensure!(

View on GitHub (pinned to 18893faf8b)

Solutions

1. Retry validation after the chain stabilizes; treat removed==true logs as reorged and re-ingest from the new canonical block.
2. Pin to a single consistent RPC node for both watermark ingestion and receipt/log fetch.
3. Confirm position.number/transaction_index and expected_block_hash come from the same block header query.
4. On a dev/local fork, reset the profiler watermark state after any chain reset.

Example fix

// before: single validation pass
validate_profiler_log(&log, &position, expected_block_hash)?;

// after: reorg-aware handling
if log.removed {
    reingest_watermark_from_canonical(&mut profiler).await?;
} else {
    validate_profiler_log(&log, &position, expected_block_hash)?;
}

if log.removed { reingest_from_canonical(); }
assert_eq!(extract_block_number(&log)?, position.number);
assert_eq!(extract_block_number(&log)?, position.number);
assert_eq!(extract_transaction_index(&log)?, position.transaction_index);

fn log_at_position(log: &Log, position: &Position) -> bool {
    !log.removed
        && extract_block_number(log).ok() == Some(position.number)
        && extract_transaction_index(log).ok() == Some(position.transaction_index)
}

match validate_profiler_log(&log, &position, &block_hash) {
    Ok(()) => Ok(()),
    Err(e) if e.to_string().contains("log position does not match") => handle_reorg_and_reingest(e),
    Err(e) => Err(e),
}

When it happens

Trigger: Profiler validation where the matched log has removed==true, or extract_block_number != position.number, extract_transaction_index != position.transaction_index, log_transaction_hash != the profiler transaction hash, or the log's block hash differs from expected_block_hash.

Common situations: Chain reorg between watermark ingestion and validation; RPC load balancer serving logs from a different node than the watermark source; forked dev chains reset between runs; out-of-sync execution node.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Replacement scan cursor conflicts with the finalized header
• Canonical nonce advanced without an authenticated signer tra
• Replacement block conflicts with its canonical header
• Replacement scan tip conflicts with the verified canonical h
• Profiler receipt position does not match its ingestion water

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/82949e20519c4c2b. Report an issue: GitHub.