linera-io/linera-protocol · critical

Retry loop exited unexpectedly: {result:?}

Error message

Retry loop exited unexpectedly: {result:?}

What it means

In serve_loop's tokio::select!, the retry loop task exited. This loop re-drives previously failed operations (e.g. settlements that timed out or reverted) on a schedule; its unexpected exit — error or panic — fails the whole relay because failed operations would never be retried.

Source

Thrown at linera-bridge/src/relay/mod.rs:514

    let mut inbox_drain_interval = tokio::time::interval(monitor_scan_interval);
    // Consume the immediate first tick — the startup drain above already ran.
    inbox_drain_interval.tick().await;

    // ── Main loop: process chain operations + notifications ──
    tracing::info!("Listening for chain operations and notifications...");
    loop {
        tokio::select! {
            result = &mut chain_listener_handle => {
                anyhow::bail!("Chain listener exited unexpectedly: {result:?}");
            }
            result = &mut evm_scan_handle => {
                anyhow::bail!("EVM scan loop exited unexpectedly: {result:?}");
            }
            result = &mut linera_scan_handle => {
                anyhow::bail!("Linera scan loop exited unexpectedly: {result:?}");
            }
            result = &mut retry_handle => {
                anyhow::bail!("Retry loop exited unexpectedly: {result:?}");
            }
            result = &mut http_server_handle => {
                anyhow::bail!("HTTP server exited unexpectedly: {result:?}");
            }
            result = &mut admin_server_handle => {
                anyhow::bail!("Admin HTTP server exited unexpectedly: {result:?}");
            }
            _ = inbox_drain_interval.tick() => {
                // Periodic safety net for a missed `NewIncomingBundle`: sync and
                // drain the inbox so stranded messages (e.g. user burns) are
                // eventually processed even without a fresh notification.
                if let Err(e) = chain_client.synchronize_from_validators().await {
                    tracing::warn!("Periodic sync before inbox drain failed: {e}");
                } else {
                    match chain_client.process_inbox().await {
                        Ok((certs, _)) if !certs.is_empty() => {
                            tracing::info!(count = certs.len(), "Periodic inbox drain processed messages");
                        }

View on GitHub (pinned to 6c226ddcb3)

Solutions

1. Find the retry loop's last logged error before this bail — it names the operation and underlying failure.
2. Fix that underlying cause (RPC availability, contract state drift, storage), then restart the relay; the retry queue is reprocessed from persisted state.
3. If a specific poisoned retry entry panics repeatedly, remove/resolve that entry (e.g. mark the deposit as failed manually) after root-causing.
4. Supervise the relay process for automatic restart.

Example fix

// before
result = &mut retry_handle => {
    anyhow::bail!("Retry loop exited unexpectedly: {result:?}");
}

// after: propagate the inner error chain instead of only Debug-printing it
result = &mut retry_handle => {
    anyhow::bail!("Retry loop exited unexpectedly: {result:?}; \
                 inspect preceding retry-loop error logs to identify the poisoned operation");
}

// Fatal by design; supervisor restart reprocesses the persisted retry queue.
// On repeated crashes, inspect the retry queue for a poisoned entry and resolve it manually.

When it happens

Trigger: The retry future returns Err (persistent failure re-loading or re-executing the retry queue, e.g. storage/RPC errors that outlive its internal handling) or panics while re-running an operation such as re-sending a settlement transaction.

Common situations: Both EVM and retry paths hitting the same broken RPC for long stretches; a retry entry whose replay panics due to ABI/state drift after a contract upgrade; storage backing the retry queue becoming unavailable.

Related errors

• Chain listener exited unexpectedly: {result:?}
• EVM scan loop exited unexpectedly: {result:?}
• Linera scan loop exited unexpectedly: {result:?}
• HTTP server exited unexpectedly: {result:?}
• Admin HTTP server exited unexpectedly: {result:?}

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/c0cd4bb028a255c8. Report an issue: GitHub.