xai-org/x-algorithm · critical · ValueError

No partitions found for topic {topic}, available topics: {av

Error message

No partitions found for topic {topic}, available topics: {available}

What it means

In the multi-consumer path, _consume_multi_consumer tries three times (initial partitions_for_topic plus two forced metadata updates) to discover the topic's partitions before fan-out; if all fail it raises with the list of topics the discovery consumer can see. It is the multi-consumer equivalent of the topic-not-found check.

Source

Thrown at phoenix/xrex/data/streaming/kafkaconsumer.py:991

        security_protocol="SASL_SSL",
        sasl_kerberos_domain_name="kafka",
        sasl_kerberos_service_name="kafka",
        sasl_mechanism=sasl_mechanism,
        sasl_plain_username=sasl_plain_username,
        sasl_plain_password=sasl_plain_password,
        ssl_context=ssl_ctx,
        request_timeout_ms=30000,
    )
    await discovery_consumer.start()
    try:
        all_partitions = discovery_consumer.partitions_for_topic(topic)
        if not all_partitions:
            all_partitions = await force_kafka_metadata_update(discovery_consumer, topic)
        if not all_partitions:
            all_partitions = await force_kafka_metadata_update(discovery_consumer, topic)
        if not all_partitions:
            available = await discovery_consumer.topics()
            raise ValueError(
                f"No partitions found for topic {topic}, available topics: {available}"
            )
    finally:
        try:
            await discovery_consumer.stop()
        except asyncio.CancelledError:
            pass

    total_partitions = len(all_partitions)
    partition_ids = _range_partitions(total_partitions, shard_index, num_shards)
    shard_partitions = [TopicPartition(topic, i) for i in partition_ids]
    if not partition_ids:
        rank_logger.warning(
            f"Shard {shard_index} received 0 partitions (total={total_partitions}, "
            f"num_shards={num_shards}).  This shard will be idle."
        )
    rank_logger.info(
        f"Discovered {total_partitions} partitions for {topic}, "

View on GitHub (pinned to 24c60942c5)

Solutions

Check the topic name against the 'available topics' in the message and fix config/environment mismatch.
Add a readiness wait: poll partitions_for_topic with backoff for 30-60s before calling consume_messages.
Pre-create topics via infra automation so consumers never race topic creation.
If persistent, verify broker connectivity and ACLs for the client principal.

Example fix

# before
await consume_messages(topic="new-topic", ...)

# after
# wait for topic to appear in metadata
async def wait_for_topic(consumer, topic, timeout=60):
    deadline = asyncio.get_event_loop().time() + timeout
    while asyncio.get_event_loop().time() < deadline:
        if await consumer.partitions_for_topic(topic):
            return
        await asyncio.sleep(2)
    raise TimeoutError(topic)
await wait_for_topic(consumer, "new-topic")
await consume_messages(topic="new-topic", ...)

Defensive patterns

Strategy: retry

Validate before calling

parts = await consumer.partitions_for_topic(topic)
if not parts:
    for _ in range(10):
        await asyncio.sleep(2)
        parts = await consumer.partitions_for_topic(topic)
        if parts:
            break

Try / catch

try:
    await consume_messages(...)
except ValueError as e:
    if "No partitions found" in str(e):
        rank_logger.error(f"Topic unavailable: {e}; retrying after backoff")
        await asyncio.sleep(30)
        raise

Prevention

Deploy topics before consumers (infra ordering).
Add readiness probes/wait-for-topic in job startup scripts.
Retry with backoff around transient metadata emptiness.

When it happens

Trigger: consume_messages entering the multi-consumer path for a topic that does not exist in cluster metadata; brokers unreachable so both forced refreshes return nothing; transient metadata propagation lag exceeding the two retries.

Common situations: Consumer starts before the topic is provisioned; wrong bootstrap_servers/environment; topic recently deleted and recreated during a migration; heavy cluster load delaying metadata propagation.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/32c40276f77d6d5b. Report an issue: GitHub.