FuelLabs/fuel-core · error · anyhow::Error

Backlog unresolved at height {current_height}: repair failed

Error message

Backlog unresolved at height {current_height}: repair failed to reach quorum

What it means

The winning block at current_height was found on fewer than quorum nodes (a partial write by a failed former leader), so unreconciled_blocks attempted repair_sub_quorum_block to re-propose it everywhere (poa.rs:877-897). The repair returned Ok(false) — the re-write did not reach quorum — and because nothing earlier in this round had been reconciled, the whole reconciliation aborts with this error.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:891

                        "Repairing sub-quorum block at height {current_height} \
                         (found on {count}/{} nodes)",
                        blocks_by_node.len()
                    );
                    match self.repair_sub_quorum_block(&block, count) {
                        Ok(true) => {
                            tracing::info!(
                                "Repair succeeded — block at height {current_height} \
                                 now has quorum"
                            );
                            reconciled.push(block);
                        }
                        Ok(false) => {
                            tracing::warn!(
                                "Repair failed to reach quorum at height \
                                 {current_height} — will retry next round"
                            );
                            if reconciled.is_empty() {
                                return Err(anyhow!(
                                    "Backlog unresolved at height {current_height}: \
                                     repair failed to reach quorum"
                                ));
                            }
                            break;
                        }
                        Err(e) => {
                            tracing::warn!(
                                "Repair error at height {current_height}: {e}"
                            );
                            if reconciled.is_empty() {
                                return Err(anyhow!(
                                    "Backlog unresolved at height {current_height}: \
                                     repair error: {e}"
                                ));
                            }
                            break;
                        }

View on GitHub (pinned to b9d4d170da)

Solutions

1. Bring at least quorum Redis nodes to a healthy writable state so the repair write can succeed next round
2. Use the 'Repair failed to reach quorum at height …' warning plus per-node write errors to find which nodes refuse
3. Rely on automatic retry — repair is retried on every reconciliation round while conditions improve
4. If nodes reject because they already hold different data at that height, inspect the stream entries and resolve the conflict deliberately

// before repair, count writable healthy nodes:
let writable = probe_writes(&nodes).await; // SET/DEL round-trip on a probe key
anyhow::ensure!(writable >= quorum, "only {writable} writable nodes; repair cannot reach quorum");

match unreconciled_blocks(next_height).await {
    Err(e) if e.to_string().contains("repair failed to reach quorum") => {
        // transient write-availability problem: repair retries next round;
        // fix the refusing nodes (memory, read-only role, HEIGHT_EXISTS conflicts)
    }
    other => other,
}

When it happens

Trigger: A previous leader wrote the block to only a minority of Redis nodes; at repair time too few nodes are healthy or too many refuse the write (HEIGHT_EXISTS with other data, memory pressure, read-only replicas) for re-propagation to reach quorum.

Common situations: Redis nodes flapping during leader transitions; write errors from maxmemory or replica state; re-promotion storms producing epoch conflicts on some nodes.

Related errors

• Cannot reconcile: only {}/{} Redis nodes responded ({} faile
• Backlog unresolved at height {current_height}: repair error:
• Timed out reading stream entries from Redis node
• Backlog unresolved at height {current_height}: stream indica
• At least one redis url is required for leader lock

AI-assisted analysis of FuelLabs/fuel-core@b9d4d170da (2026-08-16). Data as JSON: /api/errors/7557dddcd46cf05d. Report an issue: GitHub.