cocoindex-io/cocoindex · error · RuntimeError

The Python Iggy SDK does not expose per-partition high water

Error message

The Python Iggy SDK does not expose per-partition high watermarks. Pass initial_high_watermark for multi-partition topics, or consume a single-partition topic.

What it means

The Python Iggy SDK only exposes a topic-level message count, not per-partition high watermarks. The source can therefore auto-resolve the initial offset only for single-partition topics; for multi-partition topics it raises a RuntimeError telling the user to pass initial_high_watermark explicitly.

Source

Thrown at python/cocoindex/connectors/iggy/_source.py:300

        self._initial_high_watermark = initial_high_watermark
        self._watch_guard = SingleWatcherGuard("Iggy TopicStream")

    def payloads(self) -> LiveStream[bytes]:
        """View of this stream yielding each message payload as bytes."""
        return _TopicPayloadsStream(self)

    async def _resolve_initial_high_watermark(self) -> int:
        """Resolve the initial next-offset watermark for readiness."""
        if self._initial_high_watermark is not None:
            return self._initial_high_watermark

        topic = await self._client.get_topic(self._stream, self._topic)
        if topic is None:
            raise RuntimeError(
                f"Iggy topic {self._stream}/{self._topic} does not exist."
            )
        if topic.partitions_count != 1:
            raise RuntimeError(
                "The Python Iggy SDK does not expose per-partition high watermarks. "
                "Pass initial_high_watermark for multi-partition topics, or consume "
                "a single-partition topic."
            )
        return int(topic.messages_count)

    async def _create_consumer(self) -> IggyConsumer:
        """Create an Iggy consumer group configured for manual offset storage."""
        return await self._client.consumer_group(
            name=self._consumer_group,
            stream=self._stream,
            topic=self._topic,
            partition_id=self._partition_id,
            polling_strategy=PollingStrategy.Next(),
            batch_length=self._batch_length,
            auto_commit=AutoCommit.Disabled(),
            poll_interval=self._poll_interval,
            polling_retry_interval=self._polling_retry_interval,

View on GitHub (pinned to e84aa99b32)

Solutions

Pass an explicit initial_high_watermark value when constructing/consuming the topic
Use a single-partition topic so the source can resolve the watermark automatically
Partition the consumer as one TopicStream per partition with explicit offsets as advised by the connector

Example fix

// before
stream = iggy.TopicStream(client, stream="s", topic="multi_part")
// after
stream = iggy.TopicStream(client, stream="s", topic="multi_part", initial_high_watermark=12345)

Defensive patterns

Strategy: validation

Validate before calling

topic = await client.get_topic(stream_name, topic_name)
if topic is not None and topic.partitions_count != 1 and initial_high_watermark is None:
    raise ValueError("Pass initial_high_watermark for multi-partition Iggy topics")

Try / catch

try:
    await source.start()
except RuntimeError as e:
    if "initial_high_watermark" in str(e):
        configure_explicit_watermark(last_known_offset)

Prevention

Persist offsets and pass initial_high_watermark whenever topics may have >1 partition
Create consumer topics as single-partition when auto-resolution is desired
Assert partitions_count at startup and fail fast with a clear message

When it happens

Trigger: Calling _watch (via start/monitoring) on an Iggy topic whose partitions_count != 1 without supplying initial_high_watermark.

Common situations: Scaling a topic to multiple partitions after initially running single-partition; deploying the connector against a production topic with several partitions.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of cocoindex-io/cocoindex@e84aa99b32 (2026-09-08). Data as JSON: /api/errors/0da08a8be93f1fca. Report an issue: GitHub.