quickwit-oss/quickwit · error

topic `{}` does not exist

Error message

topic `{}` does not exist

What it means

During Kafka connectivity checking, metadata was fetched for the topic but the returned cluster metadata contained no topics, so Quickwit concludes the topic does not exist and the source cannot consume from it.

Source

Thrown at quickwit/quickwit-indexing/src/source/kafka_source.rs:643

pub(super) async fn check_connectivity(params: KafkaSourceParams) -> anyhow::Result<()> {
    let mut client_config = parse_client_params(params.client_params)?;

    let consumer: BaseConsumer<DefaultConsumerContext> = client_config
        .set("group.id", "quickwit-connectivity-check".to_string())
        .set_log_level(RDKafkaLogLevel::Error)
        .create()?;

    let topic = params.topic.clone();
    let timeout = Timeout::After(Duration::from_secs(5));
    let cluster_metadata = spawn_blocking(move || {
        consumer
            .fetch_metadata(Some(&topic), timeout)
            .with_context(|| format!("failed to fetch metadata for topic `{topic}`"))
    })
    .await??;

    if cluster_metadata.topics().is_empty() {
        bail!("topic `{}` does not exist", params.topic);
    }
    let topic_metadata = &cluster_metadata.topics()[0];
    assert_eq!(topic_metadata.name(), params.topic); // Belt and suspenders.

    if topic_metadata.partitions().is_empty() {
        bail!("topic `{}` has no partitions", params.topic);
    }
    Ok(())
}

/// Creates a new `KafkaSourceConsumer`.
fn create_consumer(
    index_uid: &IndexUid,
    source_id: &str,
    params: KafkaSourceParams,
    events_tx: mpsc::Sender<KafkaEvent>,
) -> anyhow::Result<(ClientConfig, RdKafkaConsumer, GroupId)> {
    // Group ID is limited to 255 characters.

View on GitHub (pinned to a39730c5cd)

Solutions

1. Verify the topic exists: `kafka-topics.sh --bootstrap-server <broker> --describe --topic <topic>` and fix the topic name in the source config.
2. Create the missing topic if it was deleted.
3. Confirm the bootstrap_servers/client_params in the source config point at the intended cluster.
4. Re-run the connectivity check: `quickwit source check-connectivity`.

Example fix

// before
source:
  params:
    topic: log-events-prod
// after (topic actually named differently)
kafka-topics.sh --bootstrap-server broker:9092 --list
source:
  params:
    topic: log-events

kafka-topics.sh --bootstrap-server $BROKERS --describe --topic $TOPIC || echo "topic $TOPIC missing"

// run connectivity check before enabling the source
match quickwit_source_check(topic, brokers).await {
    Err(e) if e.to_string().contains("does not exist") => create_or_fix_topic().await?,
    other => other?,
}

When it happens

Trigger: `check_connectivity` fetches metadata via `fetch_metadata(Some(&topic), timeout)`; the `rdkafka` response comes back with an empty `topics()` list. (Note: a real `Err` from fetch_metadata would instead surface the 'failed to fetch metadata' context error.)

Common situations: Topic deleted after the source was configured; typo in the topic name in the source config; broker auto-create disabled and topic never created; connecting to the wrong Kafka cluster.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

• topic `{}` has no partitions
• failed to parse Kafka client log level. value `{}` is not su
• failed to parse Kafka client parameters. `client_params` mus
• failed to parse Kafka client parameters. `client_params.{}`
• Quickwit was compiled without the `kafka` feature

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/aa6c4a351ef2ec4d. Report an issue: GitHub.