xai-org/x-algorithm · critical · ValueError

No partitions found for topic '{topic}'. Available topics: {

Error message

No partitions found for topic '{topic}'. Available topics: {available}

What it means

Raised by discover_partition_count when the Kafka consumer cannot find any partitions for the requested topic, even after a forced metadata refresh. The error lists all topics visible from cluster metadata so you can see what Kafka actually knows about. It almost always means the topic does not exist, is misspelled, or the broker cluster the client is connected to has no such topic.

Source

Thrown at phoenix/xrex/data/streaming/kafkaconsumer.py:458

    consumer = AIOKafkaConsumer(
        bootstrap_servers=bootstrap_servers,
        security_protocol="SASL_SSL",
        sasl_kerberos_domain_name="kafka",
        sasl_kerberos_service_name="kafka",
        sasl_mechanism=sasl_mechanism,
        sasl_plain_username=sasl_plain_username,
        sasl_plain_password=sasl_plain_password,
        ssl_context=ssl_ctx,
        request_timeout_ms=30000,
    )
    await consumer.start()
    try:
        partitions = consumer.partitions_for_topic(topic)
        if not partitions:
            partitions = await force_kafka_metadata_update(consumer, topic)
        if not partitions:
            available = await consumer.topics()
            raise ValueError(
                f"No partitions found for topic '{topic}'. Available topics: {available}"
            )
        count = len(partitions)
        rank_logger.info(f"Discovered {count} partitions for topic '{topic}' from Kafka metadata.")
        return count
    finally:
        try:
            await consumer.stop()
        except asyncio.CancelledError:
            pass


def _range_partitions(total_partitions: int, shard_index: int, num_shards: int) -> list[int]:
    if num_shards <= 0:
        raise ValueError(f"num_shards must be > 0, got {num_shards}")
    if total_partitions < 0:
        raise ValueError(f"total_partitions must be >= 0, got {total_partitions}")
    start = shard_index * total_partitions // num_shards

View on GitHub (pinned to 24c60942c5)

Solutions

Compare the topic name against the 'Available topics' list in the error message and fix typos/case in your config.
Verify you are connecting to the right cluster: check bootstrap_servers environment/flags and run kafka-topics --bootstrap-server <brokers> --list.
If the topic genuinely does not exist, create it (kafka-topics --create ...) or enable auto-create, then retry.
If the topic exists but is not listed, check ACL/permissions for the client principal and broker advertised.listeners reachability.

Example fix

# before
partitions = await discover_partition_count(consumer, "my-topc")

# after
partitions = await discover_partition_count(consumer, "my-topic")  # match 'Available topics' from error

Defensive patterns

Strategy: validation

Validate before calling

async def topic_exists(consumer, topic: str) -> bool:
    return bool(await consumer.partitions_for_topic(topic))

Try / catch

try:
    count = await discover_partition_count(consumer, topic)
except ValueError as e:
    if "No partitions found" in str(e):
        rank_logger.error(f"Topic not visible; check bootstrap_servers/topic name: {e}")
        raise

Prevention

Validate topic names against a config schema at startup.
Run a preflight kafka-topics --describe in deploy scripts.
Wait for metadata propagation after topic creation before starting consumers.

When it happens

Trigger: Calling ensure_partition_count/consumption_loop with a topic name that does not exist on the brokers pointed to by bootstrap_servers; topic auto-creation disabled (auto.create.topics.enable=false) so the producer never created it; connected to the wrong cluster/environment; ACLs hiding the topic from the client.

Common situations: Typo in the topic name in config; pointing at a staging bootstrap server while the topic only exists in production; Kafka cluster migrated and old topic names removed; topic not yet created by an infrastructure/Terraform step; brokers unreachable so metadata comes back empty.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/7ab8f7b8f78284ed. Report an issue: GitHub.