linera-io/linera-protocol · error · NodeError

MissingCertificates

MissingCertificates

Error message

NodeError::MissingCertificates(missing_hashes)

What it means

GrpcClient::download_certificates requests certificates from a validator in batches by hash, expecting honest validators to return them in request order; each round trims the requested-hash list by the number received. NodeError::MissingCertificates(missing_hashes) is raised when, after all rounds, some requested certificate hashes were still not delivered.

Source

Thrown at linera-rpc/src/grpc/client.rs:581

                missing
            )?)?
            .into_iter()
            .map(|cert| {
                ConfirmedBlockCertificate::try_from(cert)
                    .map_err(|_| NodeError::UnexpectedCertificateValue)
            })
            .collect::<Result<_, _>>()?;

            // In the case of the server not returning any certificates, we break the loop.
            if received.is_empty() {
                break;
            }

            // Honest validator should return certificates in the same order as the requested hashes.
            missing_hashes = missing_hashes[received.len()..].to_vec();
            certs_collected.append(&mut received);
        }
        ensure!(
            missing_hashes.is_empty(),
            NodeError::MissingCertificates(missing_hashes)
        );
        Ok(certs_collected)
    }

    #[instrument(target = "grpc_client", skip(self), err(level = Level::DEBUG), fields(address = self.address))]
    async fn download_certificates_by_heights(
        &self,
        chain_id: ChainId,
        heights: Vec<BlockHeight>,
    ) -> Result<Vec<ConfirmedBlockCertificate>, NodeError> {
        let mut missing = heights.into_iter().collect::<BTreeSet<_>>();
        let mut certs_collected = vec![];
        while !missing.is_empty() {
            let request = CertificatesByHeightRequest {
                chain_id,
                heights: missing.iter().copied().collect(),

View on GitHub (pinned to 6c226ddcb3)

Solutions

  1. Retry the download against a fully synced committee validator; the error payload lists exactly which hashes are still missing
  2. Wait for the validator to process those heights, then re-request only the missing hashes
  3. Confirm client and validator share the same genesis configuration (same chain, same committee)
  4. If blocks were pruned, obtain them from a validator configured for archive/long retention

Example fix

// before
let certs = client.download_certificates(&hashes).await?; // MissingCertificates from a lagging validator

// after
let certs = loop {
    match client.download_certificates(&hashes).await {
        Ok(certs) => break certs,
        Err(NodeError::MissingCertificates(missing)) if attempt < 5 => {
            tokio::time::sleep(backoff(attempt)).await; // let the validator catch up
            hashes = missing; // only re-request what is still missing
            attempt += 1;
        }
        Err(e) => return Err(e.into()),
    }
};
Defensive patterns

Strategy: retry

Try / catch

Catch NodeError::MissingCertificates, read the missing_hashes from the error, sleep a backoff interval (validators often only need time to catch up), and re-issue download_certificates with just the missing hashes; after a few failures, switch to a different, fully synced validator endpoint.

Prevention

When it happens

Trigger: Asking a validator for certificates it does not hold: hashes beyond its highest processed block height, pruned blocks, a wrong shard, or a validator returning fewer certificates than requested without an error.

Common situations: Syncing a client or worker from a validator that lags behind the requested heights; querying block hashes that do not exist on that network (mismatched genesis); aggressive pruning on validators; degraded networks returning short batches.

Understand the failure class

Related errors

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/8cae0fb468c3b9dd. Report an issue: GitHub.