linera-io/linera-protocol · critical · ChainError

InternalError

InternalError

Error message

checkpoint oracle response has {} outbox block hashes but the restored state references {} distinct heights

What it means

Internal invariant check after restoring execution state from a checkpoint blob: the number of distinct unfinalized outbox heights in the restored state must equal the number of outbox block hashes the checkpoint oracle response recorded. A mismatch means the certified checkpoint blob and its oracle response are inconsistent with each other — this is a defect or corruption, not a caller mistake.

Source

Thrown at linera-core/src/chain_worker/state.rs:1179

            self.save().await?;
            return Err(WorkerError::BlocksNotFound(missing_blocks));
        }
        self.chain
            .execution_state
            .restore_from_content(&bytes)
            .await?;
        // `restore_from_content` writes directly to storage and leaves the
        // in-memory view in an undefined state — reload from storage.
        self.chain = self.storage.load_chain(chain_id).await?;
        // Re-populate `block_hashes` for every pre-checkpoint sender block the
        // chain still needs. The heights live in the just-restored execution
        // state (`unfinalized_message_blocks`); the matching hashes are the
        // ones the producer recorded in the oracle response, certified by the
        // checkpoint cert we already trust. Without this, the next step
        // (re-executing the checkpoint to verify its outcome) would fail
        // because `collect_unfinalized_block_hashes` looks these up.
        let heights = self.chain.collect_unfinalized_heights().await?;
        ensure!(
            heights.len() == outbox_block_hashes.len(),
            ChainError::InternalError(format!(
                "checkpoint oracle response has {} outbox block hashes but the \
                 restored state references {} distinct heights",
                outbox_block_hashes.len(),
                heights.len(),
            ))
        );
        for (height, hash) in heights.into_iter().zip(outbox_block_hashes) {
            self.chain.block_hashes.insert(&height, hash)?;
        }
        // Rebuild the off-chain outbox state (queues, counters,
        // nonempty_outboxes) from the on-chain unfinalized map so that this
        // node can resume pushing pre-checkpoint messages forward. The outbox
        // isn't part of the certified checkpoint blob, so without this a
        // bootstrapped validator would silently stop delivering pending
        // messages.
        let tracked = self.tracked_full_chains();

View on GitHub (pinned to 6c226ddcb3)

Solutions

1. Report it upstream as a bug, attaching the block hash and checkpoint blob IDs
2. Re-create and re-certify the checkpoint on the producing chain
3. As a workaround, re-sync the affected chain from genesis or from a known-good checkpoint

fn is_checkpoint_invariant_error(e: &WorkerError) -> bool {
    matches!(e, WorkerError::ChainError(ref c) if matches!(**c, ChainError::InternalError(_)))
}

match client.submit_confirmed(cert).await {
    Err(e) if matches!(e, ref x if x.is_checkpoint_invariant_error()) => {
        // Invariant violation: do not retry blindly. Capture diagnostics
        // (block hash, checkpoint blob ids, versions) and report upstream;
        // re-sync the chain from a known-good checkpoint to recover.
        report_diagnostics(&cert).await;
        return Err(e);
    }
    other => other?,
}

When it happens

Trigger: process_confirmed_block on a checkpoint block whose oracle response lists outbox_block_hashes that do not correspond one-to-one with the restored state's unfinalized message-block heights.

Common situations: Should not occur in correct operation; indicates a bug in checkpoint production, a corrupted checkpoint blob, or divergent code versions between producer and validator.

Related errors

• Checkpoint precondition failed: Checkpoint must be the first
• Checkpoint precondition failed: chain has consumed system ev
• It is illegal to call function summarize_events from an oper
• CheckpointPreconditionFailed
• Unexpected response type

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/097d42915750f858. Report an issue: GitHub.