{"record":{"id":"0da08a8be93f1fca","repo":"cocoindex-io/cocoindex","slug":"the-python-iggy-sdk-does-not-expose-per-partition","errorCode":null,"errorMessage":"The Python Iggy SDK does not expose per-partition high watermarks. Pass initial_high_watermark for multi-partition topics, or consume a single-partition topic.","messagePattern":"The Python Iggy SDK does not expose per-partition high watermarks\\. Pass initial_high_watermark for multi-partition topics, or consume a single-partition topic\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"python/cocoindex/connectors/iggy/_source.py","lineNumber":300,"sourceCode":"        self._initial_high_watermark = initial_high_watermark\n        self._watch_guard = SingleWatcherGuard(\"Iggy TopicStream\")\n\n    def payloads(self) -> LiveStream[bytes]:\n        \"\"\"View of this stream yielding each message payload as bytes.\"\"\"\n        return _TopicPayloadsStream(self)\n\n    async def _resolve_initial_high_watermark(self) -> int:\n        \"\"\"Resolve the initial next-offset watermark for readiness.\"\"\"\n        if self._initial_high_watermark is not None:\n            return self._initial_high_watermark\n\n        topic = await self._client.get_topic(self._stream, self._topic)\n        if topic is None:\n            raise RuntimeError(\n                f\"Iggy topic {self._stream}/{self._topic} does not exist.\"\n            )\n        if topic.partitions_count != 1:\n            raise RuntimeError(\n                \"The Python Iggy SDK does not expose per-partition high watermarks. \"\n                \"Pass initial_high_watermark for multi-partition topics, or consume \"\n                \"a single-partition topic.\"\n            )\n        return int(topic.messages_count)\n\n    async def _create_consumer(self) -> IggyConsumer:\n        \"\"\"Create an Iggy consumer group configured for manual offset storage.\"\"\"\n        return await self._client.consumer_group(\n            name=self._consumer_group,\n            stream=self._stream,\n            topic=self._topic,\n            partition_id=self._partition_id,\n            polling_strategy=PollingStrategy.Next(),\n            batch_length=self._batch_length,\n            auto_commit=AutoCommit.Disabled(),\n            poll_interval=self._poll_interval,\n            polling_retry_interval=self._polling_retry_interval,","sourceCodeStart":282,"sourceCodeEnd":318,"githubUrl":"https://github.com/cocoindex-io/cocoindex/blob/e84aa99b3292c5270a4b313b2a7137ad9ce8ab3b/python/cocoindex/connectors/iggy/_source.py#L282-L318","documentation":"The Python Iggy SDK only exposes a topic-level message count, not per-partition high watermarks. The source can therefore auto-resolve the initial offset only for single-partition topics; for multi-partition topics it raises a RuntimeError telling the user to pass initial_high_watermark explicitly.","triggerScenarios":"Calling _watch (via start/monitoring) on an Iggy topic whose partitions_count != 1 without supplying initial_high_watermark.","commonSituations":"Scaling a topic to multiple partitions after initially running single-partition; deploying the connector against a production topic with several partitions.","solutions":["Pass an explicit initial_high_watermark value when constructing/consuming the topic","Use a single-partition topic so the source can resolve the watermark automatically","Partition the consumer as one TopicStream per partition with explicit offsets as advised by the connector"],"exampleFix":"// before\nstream = iggy.TopicStream(client, stream=\"s\", topic=\"multi_part\")\n// after\nstream = iggy.TopicStream(client, stream=\"s\", topic=\"multi_part\", initial_high_watermark=12345)","handlingStrategy":"validation","validationCode":"topic = await client.get_topic(stream_name, topic_name)\nif topic is not None and topic.partitions_count != 1 and initial_high_watermark is None:\n    raise ValueError(\"Pass initial_high_watermark for multi-partition Iggy topics\")","typeGuard":null,"tryCatchPattern":"try:\n    await source.start()\nexcept RuntimeError as e:\n    if \"initial_high_watermark\" in str(e):\n        configure_explicit_watermark(last_known_offset)","preventionTips":["Persist offsets and pass initial_high_watermark whenever topics may have >1 partition","Create consumer topics as single-partition when auto-resolution is desired","Assert partitions_count at startup and fail fast with a clear message"],"tags":["iggy","partition","watermark","streaming"],"backgroundTag":"unsupported-operation","analyzedSha":"e84aa99b3292c5270a4b313b2a7137ad9ce8ab3b","analyzedAt":"2026-09-08T15:59:19.997Z","contentChangedAt":"2026-09-08T15:59:19.997Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}