{"record":{"id":"097d42915750f858","repo":"linera-io/linera-protocol","slug":"internalerror","errorCode":"InternalError","errorMessage":"checkpoint oracle response has {} outbox block hashes but the restored state references {} distinct heights","messagePattern":"checkpoint oracle response has (.+?) outbox block hashes but the restored state references (.+?) distinct heights","errorType":"exception","errorClass":"ChainError","httpStatus":null,"severity":"critical","filePath":"linera-core/src/chain_worker/state.rs","lineNumber":1179,"sourceCode":"            self.save().await?;\n            return Err(WorkerError::BlocksNotFound(missing_blocks));\n        }\n        self.chain\n            .execution_state\n            .restore_from_content(&bytes)\n            .await?;\n        // `restore_from_content` writes directly to storage and leaves the\n        // in-memory view in an undefined state — reload from storage.\n        self.chain = self.storage.load_chain(chain_id).await?;\n        // Re-populate `block_hashes` for every pre-checkpoint sender block the\n        // chain still needs. The heights live in the just-restored execution\n        // state (`unfinalized_message_blocks`); the matching hashes are the\n        // ones the producer recorded in the oracle response, certified by the\n        // checkpoint cert we already trust. Without this, the next step\n        // (re-executing the checkpoint to verify its outcome) would fail\n        // because `collect_unfinalized_block_hashes` looks these up.\n        let heights = self.chain.collect_unfinalized_heights().await?;\n        ensure!(\n            heights.len() == outbox_block_hashes.len(),\n            ChainError::InternalError(format!(\n                \"checkpoint oracle response has {} outbox block hashes but the \\\n                 restored state references {} distinct heights\",\n                outbox_block_hashes.len(),\n                heights.len(),\n            ))\n        );\n        for (height, hash) in heights.into_iter().zip(outbox_block_hashes) {\n            self.chain.block_hashes.insert(&height, hash)?;\n        }\n        // Rebuild the off-chain outbox state (queues, counters,\n        // nonempty_outboxes) from the on-chain unfinalized map so that this\n        // node can resume pushing pre-checkpoint messages forward. The outbox\n        // isn't part of the certified checkpoint blob, so without this a\n        // bootstrapped validator would silently stop delivering pending\n        // messages.\n        let tracked = self.tracked_full_chains();","sourceCodeStart":1161,"sourceCodeEnd":1197,"githubUrl":"https://github.com/linera-io/linera-protocol/blob/6c226ddcb332ef55118dc8d0aafbd093d5420899/linera-core/src/chain_worker/state.rs#L1161-L1197","documentation":"Internal invariant check after restoring execution state from a checkpoint blob: the number of distinct unfinalized outbox heights in the restored state must equal the number of outbox block hashes the checkpoint oracle response recorded. A mismatch means the certified checkpoint blob and its oracle response are inconsistent with each other — this is a defect or corruption, not a caller mistake.","triggerScenarios":"process_confirmed_block on a checkpoint block whose oracle response lists outbox_block_hashes that do not correspond one-to-one with the restored state's unfinalized message-block heights.","commonSituations":"Should not occur in correct operation; indicates a bug in checkpoint production, a corrupted checkpoint blob, or divergent code versions between producer and validator.","solutions":["Report it upstream as a bug, attaching the block hash and checkpoint blob IDs","Re-create and re-certify the checkpoint on the producing chain","As a workaround, re-sync the affected chain from genesis or from a known-good checkpoint"],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn is_checkpoint_invariant_error(e: &WorkerError) -> bool {\n    matches!(e, WorkerError::ChainError(ref c) if matches!(**c, ChainError::InternalError(_)))\n}","tryCatchPattern":"match client.submit_confirmed(cert).await {\n    Err(e) if matches!(e, ref x if x.is_checkpoint_invariant_error()) => {\n        // Invariant violation: do not retry blindly. Capture diagnostics\n        // (block hash, checkpoint blob ids, versions) and report upstream;\n        // re-sync the chain from a known-good checkpoint to recover.\n        report_diagnostics(&cert).await;\n        return Err(e);\n    }\n    other => other?,\n}","preventionTips":["Pin producer and validator to compatible versions when checkpoints are involved","Verify checkpoint blob integrity (hashes) when transferring them between nodes","Treat any InternalError from checkpoint restore as a defect to report, not a transient condition"],"tags":["checkpoint","internal-error","invariant","state-restore","linera"],"backgroundTag":"internal-invariant-violation","analyzedSha":"6c226ddcb332ef55118dc8d0aafbd093d5420899","analyzedAt":"2026-08-22T22:49:09.787Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}