{"record":{"id":"c0271026457168e5","repo":"xai-org/x-algorithm","slug":"no-healthy-kafka-region-available-for-topic-self","errorCode":null,"errorMessage":"No healthy Kafka region available for topic {self.topic!r}","messagePattern":"No healthy Kafka region available for topic (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"critical","filePath":"grox/libs/kafka_cli/multi_region_producer.py","lineNumber":134,"sourceCode":"            try:\n                await producer.stop()\n            except Exception:\n                logger.exception(f\"Failed to stop producer for region {region!r}\")\n        self._producers = {}\n\n    def _candidate_regions(self) -> list[str]:\n        candidates = [\n            region for region in self.config.clusters if region in self._producers\n        ]\n        random.shuffle(candidates)\n        return candidates\n\n    async def send(self, id: str, value: bytes):\n        if not self._producers:\n            raise RuntimeError(\"Producer not started\")\n        candidates = self._candidate_regions()\n        if not candidates:\n            raise RuntimeError(\n                f\"No healthy Kafka region available for topic {self.topic!r}\"\n            )\n        start = time.perf_counter()\n        errors: list[BaseException] = []\n        for region in candidates:\n            attributes = {\"topic\": self.topic, \"region\": region}\n            try:\n                await self._producers[region].send_and_wait(\n                    self.topic, key=id.encode(), value=value\n                )\n                Metrics.counter(\"kafka_producer.sent.count\").add(\n                    1, attributes=attributes\n                )\n                Metrics.histogram(\"kafka_producer.send_duration\").record(\n                    time.perf_counter() - start, attributes=attributes\n                )\n                return\n            except Exception as e:","sourceCodeStart":116,"sourceCodeEnd":152,"githubUrl":"https://github.com/xai-org/x-algorithm/blob/24c60942c5c5fdad3a6addffb4c6e6d2f228f04f/grox/libs/kafka_cli/multi_region_producer.py#L116-L152","documentation":"send() picks candidate regions via _candidate_regions() (which filters by health/circuit-breaker state and shuffles). If every configured region is currently marked unhealthy, there is nothing to attempt and it raises RuntimeError naming the topic. This is the all-regions-down aggregate signal, raised before any send attempt.","triggerScenarios":"All regions' producers are in an open circuit/failed state (recent send or connection failures exceeded thresholds) when send() is called; also possible right after start if health initialization marked everything down.","commonSituations":"Kafka outage or network partition affecting every region; mTLS/credential failure common to all clusters; aggressive circuit-breaker settings that never half-open to retry; a bad topic name causing consistent metadata errors that trip breakers everywhere.","solutions":["Check broker reachability and credentials for every region (the breakers tripped for a reason — inspect logs for the underlying send/connection errors).","Wait for the recovery/retry interval so circuit breakers half-open and retry, or restart the producer to force reconnection.","If breakers are too aggressive, tune their thresholds/intervals so transient failures do not blackhole all regions.","Ensure the topic exists in at least one region and the client has permissions for it."],"exampleFix":"# before\ntry:\n    await producer.send('orders', b'x')\nexcept RuntimeError as e:  # No healthy Kafka region available\n    drop_record()  # data loss\n\n# after\ntry:\n    await producer.send('orders', b'x')\nexcept RuntimeError:\n    await outbox.buffer('orders', b'x')  # retry later\n    raise","handlingStrategy":"fallback","validationCode":"# proactive: expose/track producer health before sending\ncandidates = producer._candidate_regions()\nif not candidates:\n    route_to_outboxInstead(topic, value)  # buffer for retry","typeGuard":null,"tryCatchPattern":"try:\n    await producer.send(k, v)\nexcept RuntimeError as e:\n    if 'No healthy Kafka region' in str(e):\n        await outbox.buffer(k, v)\n        alert_ops('kafka-all-regions-down')\n        raise\n    raise","preventionTips":["Watch circuit-breaker metrics per region so you see degradation before total loss","Keep an outbox/dead-letter buffer for durability during outages","Alert on consecutive region failures, not just the final all-down error"],"tags":["kafka","circuit-breaker","outage","producer","availability"],"backgroundTag":"no-healthy-upstream","analyzedSha":"24c60942c5c5fdad3a6addffb4c6e6d2f228f04f","analyzedAt":"2026-08-28T11:40:14.686Z","schemaVersion":2},"datasetVersion":"2026-08-28T16:17:29.566Z"}