What causes "consumer was dropped" in quickwit-oss/quickwit?

Look upstream in the logs for the consumer task's panic or exit cause and fix that root error (auth, broker connectivity, deserialization).

quickwit-oss/quickwit · error · ActorExitStatus

consumer was dropped

Error message

consumer was dropped

What it means

The Kafka source actor listens on an `events_rx` channel fed by a background consumer loop. `recv()` returning `None` means the sender (consumer) side was dropped, so no further Kafka events can arrive; the actor converts this into a fatal `ActorExitStatus` with this message.

Source

Thrown at quickwit/quickwit-indexing/src/source/kafka_source.rs:468

            .send_publish_lock(NewPublishLock(publish_lock), ctx)
            .await?;
        Ok(())
    }

    async fn emit_batches(
        &mut self,
        source_sink: &SourceSink,
        ctx: &SourceContext,
    ) -> Result<Duration, ActorExitStatus> {
        let now = Instant::now();
        let mut batch_builder = BatchBuilder::new(SourceType::Kafka);
        let deadline = time::sleep(*EMIT_BATCHES_TIMEOUT);
        tokio::pin!(deadline);

        loop {
            tokio::select! {
                event_opt = self.events_rx.recv() => {
                    let event = event_opt.ok_or_else(|| ActorExitStatus::from(anyhow!("consumer was dropped")))?;
                    match event {
                        KafkaEvent::Message(message) => self.process_message(message, &mut batch_builder).await?,
                        KafkaEvent::AssignPartitions { partitions, assignment_tx} => self.process_assign_partitions(ctx, &partitions, assignment_tx).await?,
                        KafkaEvent::RevokePartitions { ack_tx } => self.process_revoke_partitions(ctx, source_sink, &mut batch_builder, ack_tx).await?,
                        KafkaEvent::PartitionEOF(partition) => self.process_partition_eof(partition),
                        KafkaEvent::Error(error) => Err(ActorExitStatus::from(error))?,
                    }
                    if batch_builder.num_bytes >= BATCH_NUM_BYTES_LIMIT {
                        break;
                    }
                }
                _ = &mut deadline => {
                    break;
                }
            }
            ctx.record_progress();
        }
        if !batch_builder.checkpoint_delta.is_empty() {

View on GitHub (pinned to a39730c5cd)

Solutions

Look upstream in the logs for the consumer task's panic or exit cause and fix that root error (auth, broker connectivity, deserialization).
Restart the source/indexer so the consumer and actor are re-spawned together.
Ensure broker connectivity and credentials (`bootstrap_servers`, SASL/TLS settings) are valid so the consumer loop survives.
If caused by shutdown ordering, this is benign during stop; otherwise report as a lifecycle bug.

Defensive patterns

Strategy: retry

Try / catch

match result {
    Err(e) if e.to_string().contains("consumer was dropped") => {
        // find consumer task's root failure in logs, then restart the source actor
    }
    other => other?,
}

Prevention

Validate Kafka credentials, TLS, and bootstrap server settings before deploying.
Watch for consumer-task panics (deserialization, OOM) in logs.
Ensure graceful shutdown drops the actor before/with the consumer to distinguish intentional stops.
Alert on repeated occurrences — each means a consumer died unexpectedly.

When it happens

Trigger: `emit_batches` selects on `self.events_rx.recv()` and the channel is closed because the Kafka consumer task/thread owning the sender exited or was dropped — e.g. consumer loop panicked, spawning failed, or the consumer was torn down while the actor kept running.

Common situations: Kafka consumer thread panic (deserialization error, broker auth crash), shutdown ordering where the consumer is dropped before the actor, or resource exhaustion killing the consumer task.

Related errors

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/5c2e4195bce20b1b. Report an issue: GitHub.