quickwit-oss/quickwit · error

actor `{}` is disconnected

Error message

actor `{}` is disconnected

What it means

This error is emitted by generated code (quickwit-codegen) in the `check_connectivity` method of the mailbox-based client adapter for a gRPC service backed by an actor. When the underlying actor (e.g. the indexing service actor) is disconnected — its mailbox has been stopped or dropped — any call routed through this adapter cannot be delivered. The library throws it eagerly via `check_connectivity` so callers can detect the dead actor before issuing the actual request.

Source

Thrown at quickwit/quickwit-codegen/src/codegen.rs:1111

                let tower_svc_stack = #tower_svc_stack_name {
                    inner: inner_client,
                    #(#svc_attribute_idents),*
                };
                #client_name::new(tower_svc_stack)
            }
        }
    }
}

fn generate_tower_mailbox(context: &CodegenContext) -> TokenStream {
    let service_name = &context.service_name;
    let mailbox_name = &context.mailbox_name;
    let error_type = &context.error_type;
    let extra_mailbox_methods = if context.generate_extra_service_methods {
        quote! {
            async fn check_connectivity(&self) -> anyhow::Result<()> {
                if self.inner.is_disconnected() {
                    anyhow::bail!("actor `{}` is disconnected", self.inner.actor_instance_id())
                }
                Ok(())
            }

            fn endpoints(&self) -> Vec<quickwit_common::uri::Uri> {
                vec![quickwit_common::uri::Uri::from_str(&format!("actor://localhost/{}", self.inner.actor_instance_id())).expect("URI should be valid")]
            }
        }
    } else {
        TokenStream::new()
    };

    let (mailbox_bounds, mailbox_methods) = generate_mailbox_bounds_and_methods(context);

    quote! {
        #[derive(Debug, Clone)]
        struct MailboxAdapter<A: quickwit_actors::Actor, E> {
            inner: quickwit_actors::Mailbox<A>,

View on GitHub (pinned to a39730c5cd)

Solutions

1. Check the actor's state/logs to determine why it stopped, and restart the actor (or the node) before retrying service calls.
2. Call `check_connectivity()` before issuing operations and treat the error as 'actor must be respawned', rebuilding the service handle after restart.
3. If the actor is expected to be ephemeral, wrap client calls in retry logic that re-resolves/re-creates the actor handle after observing this error.
4. In tests, ensure the actor universe is kept alive for the duration of the calls (hold the JoinHandle/supervisor).

Example fix

// before: calling a service whose actor was stopped
let resp = indexing_service.check_connectivity().await?;
// after: verify and respawn the actor when disconnected
if indexing_service.check_connectivity().await.is_err() {
    let (_mailbox, actor_handle) = IndexingService::new(...).spawn();
    // rebuild the client from the new mailbox before calling
}

// Before issuing calls, check actor liveness
if service_actor_mailbox.instance().is_disconnected() {
    // restart actor or fail fast before making the call
}

match client.check_connectivity().await {
    Ok(()) => { /* proceed with request */ }
    Err(e) if e.to_string().contains("is disconnected") => {
        // respawn actor, rebuild client handle, then retry
    }
    Err(e) => return Err(e),
}

When it happens

Trigger: Calling `check_connectivity()` on a generated mailbox client (e.g. IndexingService's MailboxIndexingService) after the target actor has been killed, exited with an error, or its ActorContext was stopped; or any service call that internally performs the connectivity check against a disconnected actor instance.

Common situations: Indexing pipeline actor crashed during startup, a node was shut down or restarted while clients still hold the mailbox handle, actor supervision was not restarted, or tests stopped the actor universe before making a final call.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• no server currently available
• observation stream ended
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF
• failed to send cancel command to sequencer: it is probably d

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/184846295a904b98. Report an issue: GitHub.