linera-io/linera-protocol · critical

Chain listener exited unexpectedly: {result:?}

Error message

Chain listener exited unexpectedly: {result:?}

What it means

The relay's serve_loop supervises six concurrent tasks with tokio::select!. The chain-listener task (which subscribes to notifications from validators) terminated — normally or by panic — and the relay deliberately fails fast: any supervised task exiting brings down the whole serve loop, because the relay cannot do its job with a dead notification stream.

Source

Thrown at linera-bridge/src/relay/mod.rs:505

        %bridge_app_id,
        %fungible_app_id,
        "Relay is ready"
    );

    // Safety-net inbox drain. Notification delivery can be missed (relay down,
    // stream hiccup), and a missed `NewIncomingBundle` would otherwise strand
    // its messages until the next one. Reuse the monitor's scan interval as the
    // drain cadence; an empty inbox makes `process_inbox` a cheap no-op.
    let mut inbox_drain_interval = tokio::time::interval(monitor_scan_interval);
    // Consume the immediate first tick — the startup drain above already ran.
    inbox_drain_interval.tick().await;

    // ── Main loop: process chain operations + notifications ──
    tracing::info!("Listening for chain operations and notifications...");
    loop {
        tokio::select! {
            result = &mut chain_listener_handle => {
                anyhow::bail!("Chain listener exited unexpectedly: {result:?}");
            }
            result = &mut evm_scan_handle => {
                anyhow::bail!("EVM scan loop exited unexpectedly: {result:?}");
            }
            result = &mut linera_scan_handle => {
                anyhow::bail!("Linera scan loop exited unexpectedly: {result:?}");
            }
            result = &mut retry_handle => {
                anyhow::bail!("Retry loop exited unexpectedly: {result:?}");
            }
            result = &mut http_server_handle => {
                anyhow::bail!("HTTP server exited unexpectedly: {result:?}");
            }
            result = &mut admin_server_handle => {
                anyhow::bail!("Admin HTTP server exited unexpectedly: {result:?}");
            }
            _ = inbox_drain_interval.tick() => {
                // Periodic safety net for a missed `NewIncomingBundle`: sync and

View on GitHub (pinned to 6c226ddcb3)

Solutions

1. Read the relay logs immediately before this line — the actual cause (panic backtrace or stream-end warning) is logged by the dying task or an earlier warn.
2. If it is Err(JoinError) with is_panic(), fix the panic using the backtrace; if it is Ok(()), find why the stream ended (validator downtime, auth).
3. Restart the relay process (systemd/docker restart policy) — serve_loop is designed to be restarted wholesale.
4. If validator downtime is routine in your deployment, ensure restarts are automated and monitor relay liveness.

Example fix

// before (serve_loop)
result = &mut chain_listener_handle => {
    anyhow::bail!("Chain listener exited unexpectedly: {result:?}");
}

// after: enrich with panic/stream-end distinction for operators
result = &mut chain_listener_handle => {
    match result {
        Ok(()) => anyhow::bail!("Chain listener stream ended (validators unreachable?)"),
        Err(join) if join.is_panic() => anyhow::bail!("Chain listener panicked: {join}"),
        Err(join) => anyhow::bail!("Chain listener task aborted: {join}"),
    }
}

// serve_loop's error is fatal by design; handle at process level:
// systemd unit: [Service] Restart=always RestartSec=5
// in code, the supervisor task can distinguish causes:
if let Err(join) = &result && join.is_panic() {
    tracing::error!(backtrace = ?join, "chain listener panicked");
}

When it happens

Trigger: The chain listener future returns (notification stream ended, client shut down) or panics (bug in notification handling); validator connections all drop in a way that ends the stream; the wallet/client inside the listener hits a fatal error. 'result' in the message is Result<(), JoinError>-shaped: Ok means clean exit, Err(JoinError) means panic/abort.

Common situations: Validators restarting or being unavailable long enough that the notification stream closes; a panic in a code path only reachable with specific notifications; client key/permission errors killing the listener at startup.

Related errors

• EVM scan loop exited unexpectedly: {result:?}
• Linera scan loop exited unexpectedly: {result:?}
• Retry loop exited unexpectedly: {result:?}
• HTTP server exited unexpectedly: {result:?}
• Admin HTTP server exited unexpectedly: {result:?}

AI-assisted analysis of linera-io/linera-protocol@6c226ddcb3 (2026-08-22). Data as JSON: /api/errors/ed37cafd95a5f770. Report an issue: GitHub.