FuelLabs/fuel-core · error · anyhow::Error

Backlog unresolved at height {current_height}: no winning bl

Error message

Backlog unresolved at height {current_height}: no winning block candidate

What it means

Defensive branch in unreconciled_blocks (crates/fuel-core/src/service/adapters/consensus_module/poa.rs:914): nodes_with_height was > 0, so some node reported the height, yet folding the per-height entries into block_id-keyed votes produced no candidate at all; with an empty reconciled list the round aborts. Since every stored entry contributes a vote, this fires only when the height's entries were effectively emptied upstream — most plausibly all its payloads failed postcard deserialization and were skipped with a warning.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:914

                            break;
                        }
                        Err(e) => {
                            tracing::warn!(
                                "Repair error at height {current_height}: {e}"
                            );
                            if reconciled.is_empty() {
                                return Err(anyhow!(
                                    "Backlog unresolved at height {current_height}: \
                                     repair error: {e}"
                                ));
                            }
                            break;
                        }
                    }
                }
            } else {
                if reconciled.is_empty() {
                    return Err(anyhow!(
                        "Backlog unresolved at height {current_height}: \
                         no winning block candidate"
                    ));
                }
                break;
            }

            let Some(next) = current_height.checked_add(1) else {
                break;
            };
            current_height = next;
        }

        Ok(reconciled)
    }

    async fn can_produce_block(&self) -> anyhow::Result<bool> {
        tracing::debug!("Checking Redis leader lock");

View on GitHub (pinned to b9d4d170da)

Solutions

Look for 'Skipping stream entry' warnings at the same height — they carry the real deserialization cause
Align all fuel-core nodes sharing the lease_key/stream to one version
Remove or rebuild the corrupt stream entries (operator action via XDEL) so the height regains a votable candidate

Defensive patterns

Strategy: retry

Validate before calling

// operator check for the usual root cause (all payloads at a height skipped):
// grep node logs for 'Skipping stream entry: failed to deserialize block at height H'
// if present, that height has no votable block on any healthy node

Try / catch

match unreconciled_blocks(next_height).await {
    Err(e) if e.to_string().contains("no winning block candidate") => {
        // almost always downstream of deserialization skips: fix version skew /
        // corrupt entries (XDEL), then let the next round re-vote the height
    }
    other => other,
}

Prevention

Pin every node sharing a lease_key to the same fuel-core version
Alert on 'Skipping stream entry' warnings — they are the early signal of this failure
Validate restored Redis data by deserializing a sample of stream payloads before promoting it

When it happens

Trigger: All entries at current_height were dropped during read_stream_entries_on_node deserialization (logged as 'Skipping stream entry: failed to deserialize block at height …'), leaving a height key present on nodes but no votable block; or an internal invariant break in vote grouping.

Common situations: Version skew: a differently-serialized SealedBlock format written into the stream by another fuel-core version; corrupted payloads after a Redis restore.

Related errors

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/84afd441e6eb8ca9. Report an issue: GitHub.