{"record":{"id":"82949e20519c4c2b","repo":"nautechsystems/nautilus_trader","slug":"profiler-log-position-does-not-match-its-ingestion","errorCode":null,"errorMessage":"Profiler log position does not match its ingestion watermark","messagePattern":"Profiler log position does not match its ingestion watermark","errorType":"validation","errorClass":"anyhow::Error","httpStatus":null,"severity":"error","filePath":"crates/adapters/blockchain/src/execution/client.rs","lineNumber":4226,"sourceCode":"    let matching_logs = receipt\n        .logs\n        .iter()\n        .filter(|log| rpc_log::extract_log_index(log).ok() == Some(position.log_index))\n        .collect::<Vec<_>>();\n    anyhow::ensure!(\n        matching_logs.len() == 1,\n        \"Profiler receipt contains {} logs at global index {}; expected exactly one\",\n        matching_logs.len(),\n        position.log_index\n    );\n    let log = matching_logs[0];\n    let log_transaction_hash = B256::from_str(&rpc_log::extract_transaction_hash(log)?)\n        .with_context(|| \"Invalid profiler log transaction hash\")?;\n    let log_block_hash = log\n        .block_hash\n        .as_deref()\n        .ok_or_else(|| anyhow::anyhow!(\"Profiler log has no block hash\"))?;\n    anyhow::ensure!(\n        !log.removed\n            && log_transaction_hash == transaction_hash\n            && rpc_log::extract_block_number(log)? == position.number\n            && rpc_log::extract_transaction_index(log)? == position.transaction_index\n            && B256::from_str(log_block_hash)? == expected_block_hash,\n        \"Profiler log position does not match its ingestion watermark\"\n    );\n    anyhow::ensure!(\n        rpc_log::extract_address(log)? == pool_address,\n        \"Profiler watermark log did not come from expected pool {pool_address}\"\n    );\n    let signature = log\n        .topics\n        .first()\n        .ok_or_else(|| anyhow::anyhow!(\"Profiler watermark log has no event signature\"))?;\n    let supported =\n        profiler_event_signatures(pool).any(|expected| expected.eq_ignore_ascii_case(signature));\n    anyhow::ensure!(","sourceCodeStart":4208,"sourceCodeEnd":4244,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/adapters/blockchain/src/execution/client.rs#L4208-L4244","documentation":"After locating the watermark log, the library verifies the log's removed flag is false and its block number, transaction index, transaction hash, and block hash match the ingestion watermark. This error means the log's on-chain position disagrees with the recorded watermark position — usually the result of a reorg or cross-node inconsistency.","triggerScenarios":"Profiler validation where the matched log has removed==true, or extract_block_number != position.number, extract_transaction_index != position.transaction_index, log_transaction_hash != the profiler transaction hash, or the log's block hash differs from expected_block_hash.","commonSituations":"Chain reorg between watermark ingestion and validation; RPC load balancer serving logs from a different node than the watermark source; forked dev chains reset between runs; out-of-sync execution node.","solutions":["Retry validation after the chain stabilizes; treat removed==true logs as reorged and re-ingest from the new canonical block.","Pin to a single consistent RPC node for both watermark ingestion and receipt/log fetch.","Confirm position.number/transaction_index and expected_block_hash come from the same block header query.","On a dev/local fork, reset the profiler watermark state after any chain reset."],"exampleFix":"// before: single validation pass\nvalidate_profiler_log(&log, &position, expected_block_hash)?;\n\n// after: reorg-aware handling\nif log.removed {\n    reingest_watermark_from_canonical(&mut profiler).await?;\n} else {\n    validate_profiler_log(&log, &position, expected_block_hash)?;\n}","handlingStrategy":"validation","validationCode":"if log.removed { reingest_from_canonical(); }\nassert_eq!(extract_block_number(&log)?, position.number);\nassert_eq!(extract_block_number(&log)?, position.number);\nassert_eq!(extract_transaction_index(&log)?, position.transaction_index);","typeGuard":"fn log_at_position(log: &Log, position: &Position) -> bool {\n    !log.removed\n        && extract_block_number(log).ok() == Some(position.number)\n        && extract_transaction_index(log).ok() == Some(position.transaction_index)\n}","tryCatchPattern":"match validate_profiler_log(&log, &position, &block_hash) {\n    Ok(()) => Ok(()),\n    Err(e) if e.to_string().contains(\"log position does not match\") => handle_reorg_and_reingest(e),\n    Err(e) => Err(e),\n}","preventionTips":["Track reorg depth and re-ingest watermarks on removed logs","Use one consistent node for ingestion and validation","Reset watermark state after dev-chain resets"],"tags":["blockchain","reorg","consistency","logs"],"backgroundTag":"internal-invariant-violation","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}