quickwit-oss/quickwit · error

ingester service should be initialized

Error message

ingester service should be initialized

What it means

`build_ingester_service` reached the self-node branch but the optional ingester service instance was `None`, violating the internal invariant that a node listing itself as an ingester must have built and injected the local service. This expect is a programming/configuration wiring error, not a user-input error.

Solutions

1. Ensure the `ingester` service is enabled for this node (enabled_services / indexer/ingester split)
2. Ensure the ingester service was successfully built before serve_quickwit assembles the cluster changes
3. Report as a bug if the ingester service is enabled and initialized yet the expect still fires

When it happens

Trigger: Thrown at quickwit/quickwit-serve/src/lib.rs:1301 when the library encounters an invalid state.

Common situations: See trigger scenarios.

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/4d8a305b0b9f78ec. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-serve/src/lib.rs:1301

        generation_id = %chitchat_id.generation_id,
        "removing node `{}` from ingester pool",
        chitchat_id.node_id,
    );
    let node_id: NodeId = node.node_id.clone();
    Change::Remove(node_id)
}

fn build_ingester_service(
    node: &ClusterNode,
    ingester_opt: Option<impl IngesterService>,
    max_message_size: ByteSize,
    grpc_compression_encoding_opt: Option<CompressionEncoding>,
) -> IngesterServiceClient {
    if node.is_self_node() {
        // Here, since the service is available locally, we bypass the network stack
        // and use the instance directly. However, we still want client-side
        // metrics, so we use both metrics layers.
        let ingester = ingester_opt.expect("ingester service should be initialized");
        let service = ingester_service_layer_stack(
            IngesterServiceClient::tower().stack_layer(INGEST_GRPC_CLIENT_METRICS_LAYER.clone()),
        )
        .build(ingester);
        return service;
    }
    IngesterServiceClient::tower()
        .stack_layer(INGEST_GRPC_CLIENT_METRICS_LAYER.clone())
        .stack_layer(TimeoutLayer::new(GRPC_INGESTER_SERVICE_TIMEOUT))
        .build_from_channel(
            node.grpc_advertise_addr,
            node.channel(),
            max_message_size,
            grpc_compression_encoding_opt,
        )
}

async fn setup_searcher(

View on GitHub (pinned to a39730c5cd)