xai-org/x-algorithm · critical · RuntimeError

No healthy Kafka region available for topic {self.topic!r}

Error message

No healthy Kafka region available for topic {self.topic!r}

What it means

send() picks candidate regions via _candidate_regions() (which filters by health/circuit-breaker state and shuffles). If every configured region is currently marked unhealthy, there is nothing to attempt and it raises RuntimeError naming the topic. This is the all-regions-down aggregate signal, raised before any send attempt.

Source

Thrown at grox/libs/kafka_cli/multi_region_producer.py:134

            try:
                await producer.stop()
            except Exception:
                logger.exception(f"Failed to stop producer for region {region!r}")
        self._producers = {}

    def _candidate_regions(self) -> list[str]:
        candidates = [
            region for region in self.config.clusters if region in self._producers
        ]
        random.shuffle(candidates)
        return candidates

    async def send(self, id: str, value: bytes):
        if not self._producers:
            raise RuntimeError("Producer not started")
        candidates = self._candidate_regions()
        if not candidates:
            raise RuntimeError(
                f"No healthy Kafka region available for topic {self.topic!r}"
            )
        start = time.perf_counter()
        errors: list[BaseException] = []
        for region in candidates:
            attributes = {"topic": self.topic, "region": region}
            try:
                await self._producers[region].send_and_wait(
                    self.topic, key=id.encode(), value=value
                )
                Metrics.counter("kafka_producer.sent.count").add(
                    1, attributes=attributes
                )
                Metrics.histogram("kafka_producer.send_duration").record(
                    time.perf_counter() - start, attributes=attributes
                )
                return
            except Exception as e:

View on GitHub (pinned to 24c60942c5)

Solutions

Check broker reachability and credentials for every region (the breakers tripped for a reason — inspect logs for the underlying send/connection errors).
Wait for the recovery/retry interval so circuit breakers half-open and retry, or restart the producer to force reconnection.
If breakers are too aggressive, tune their thresholds/intervals so transient failures do not blackhole all regions.
Ensure the topic exists in at least one region and the client has permissions for it.

Example fix

# before
try:
    await producer.send('orders', b'x')
except RuntimeError as e:  # No healthy Kafka region available
    drop_record()  # data loss

# after
try:
    await producer.send('orders', b'x')
except RuntimeError:
    await outbox.buffer('orders', b'x')  # retry later
    raise

Defensive patterns

Strategy: fallback

Validate before calling

# proactive: expose/track producer health before sending
candidates = producer._candidate_regions()
if not candidates:
    route_to_outboxInstead(topic, value)  # buffer for retry

Try / catch

try:
    await producer.send(k, v)
except RuntimeError as e:
    if 'No healthy Kafka region' in str(e):
        await outbox.buffer(k, v)
        alert_ops('kafka-all-regions-down')
        raise
    raise

Prevention

Watch circuit-breaker metrics per region so you see degradation before total loss
Keep an outbox/dead-letter buffer for durability during outages
Alert on consecutive region failures, not just the final all-down error

When it happens

Trigger: All regions' producers are in an open circuit/failed state (recent send or connection failures exceeded thresholds) when send() is called; also possible right after start if health initialization marked everything down.

Common situations: Kafka outage or network partition affecting every region; mTLS/credential failure common to all clusters; aggressive circuit-breaker settings that never half-open to retry; a bad topic name causing consistent metadata errors that trip breakers everywhere.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/c0271026457168e5. Report an issue: GitHub.