FuelLabs/fuel-core · error · anyhow::Error

publish abandoned: quorum reached before this node responded

Error message

publish abandoned: quorum reached before this node responded (background task still running)

What it means

Informational short-circuit in publish_block_on_all_nodes: while draining per-node write results, quorum_reached(written_count) became true before every Redis node had reported. The loop breaks early instead of waiting for the remaining writes; the straggler tokio tasks keep running in the background and their channel sends become no-ops once rx is dropped. It fires on the happy path whenever quorum lands before the slowest nodes respond, not on a write failure.

Source

Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:1209

            if self.quorum_reached(written_count) {
                poa_metrics().observe_quorum_latency(
                    QuorumOp::Publish,
                    quorum_start.elapsed().as_secs_f64(),
                );
                // Short-circuit: stop draining, but leave the remaining
                // tasks running. They will each complete their write
                // attempt; when `rx` drops below, their subsequent sends
                // become no-ops, but the publish has landed on the node.
                break;
            }
        }
        drop(rx);

        results
            .into_iter()
            .map(|r| {
                r.unwrap_or_else(|| {
                    Err(anyhow!(
                        "publish abandoned: quorum reached before this \
                         node responded (background task still running)"
                    ))
                })
            })
            .collect()
    }

    /// Repropose a sub-quorum block to all Redis nodes to reach quorum.
    /// Called during reconciliation when a block exists on some nodes but
    /// below quorum — possibly from a leader that published and committed
    /// locally but whose write only reached a subset of nodes.
    ///
    /// `pre_existing_count` is the number of nodes already confirmed to
    /// have this specific block during the reconciliation read phase.
    ///
    /// Uses `publish_block_on_all_nodes` which runs `write_block.lua`:
    /// - Written: node accepted the block (counted toward quorum)

View on GitHub (pinned to add100d30d)

Solutions

1. Treat as expected behavior: quorum semantics allow ignoring slow minority nodes
2. Ensure the spawned tasks hold an Arc to the block payload so background completion is safe and allocation-free
3. Observe quorum latency metrics already emitted here to detect chronically slow nodes
4. If background task leakage matters, add a JoinHandle abort or a completion deadline after quorum is reached

When it happens

Trigger: Thrown at crates/fuel-core/src/service/adapters/consensus_module/poa.rs:1209 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of FuelLabs/fuel-core@add100d30d (2026-09-05). Data as JSON: /api/errors/e726309bcb275c21. Report an issue: GitHub.