pathwaycom/pathway · error · ValueError

'autocommit_duration_ms' must be positive; got {autocommit_d

Error message

'autocommit_duration_ms' must be positive; got {autocommit_duration_ms}. It is the maximum time between two commits and zero/negative values would prevent commits from happening.

What it means

autocommit_duration_ms in pw.io.kafka.read sets the maximum interval between consumer offset commits. Zero or negative values would mean commits never happen, so after a restart the consumer would re-read from its last persisted position (or the beginning) — Pathway rejects such values up front.

Source

Thrown at python/pathway/io/kafka/__init__.py:282

    if max_backlog_size is not None and max_backlog_size <= 0:
        raise ValueError(
            f"'max_backlog_size' must be positive; got {max_backlog_size}. "
            f"A non-positive value would prevent any entry from being "
            f"processed and the reader would never make progress."
        )
    if parallel_readers is not None and parallel_readers <= 0:
        raise ValueError(
            f"'parallel_readers' must be positive; got {parallel_readers}."
        )
    if start_from_timestamp_ms is not None and start_from_timestamp_ms < 0:
        raise ValueError(
            f"'start_from_timestamp_ms' must be non-negative; got "
            f"{start_from_timestamp_ms}. The value is a Unix timestamp in "
            f"milliseconds — negative values are pre-epoch and not "
            f"meaningful for Kafka."
        )
    if autocommit_duration_ms is not None and autocommit_duration_ms <= 0:
        raise ValueError(
            f"'autocommit_duration_ms' must be positive; got "
            f"{autocommit_duration_ms}. It is the maximum time between "
            f"two commits and zero/negative values would prevent commits "
            f"from happening."
        )

    # When 'start_from_timestamp_ms' is set, the engine seeks lazily after
    # the consumer is positioned at the partition's earliest offset, so any
    # user-supplied 'auto.offset.reset' value that doesn't already mean
    # "start at the beginning" is silently rewritten on the Rust side.
    # Surface that rewrite explicitly so somebody who picked 'latest' on
    # purpose doesn't see Pathway read from the beginning instead. The
    # librdkafka aliases 'earliest', 'beginning' and 'smallest' all mean
    # "start at the beginning" and therefore don't trigger the override.
    _START_FROM_BEGINNING_ALIASES = {"earliest", "beginning", "smallest"}
    user_offset_reset = rdkafka_settings.get("auto.offset.reset")
    if (
        start_from_timestamp_ms is not None

View on GitHub (pinned to fa2f74a464)

Solutions

Use a positive millisecond value, e.g. autocommit_duration_ms=10000 for commits at least every 10 seconds.
If you don't need to tune committing, omit the parameter and let the default apply.
Never use 0 to mean 'off' — there is no supported way to disable commits entirely via this argument.

Example fix

# before
t = pw.io.kafka.read(rdkafka_settings, topic="t", autocommit_duration_ms=0)
# after
t = pw.io.kafka.read(rdkafka_settings, topic="t", autocommit_duration_ms=10_000)

Defensive patterns

Strategy: validation

Validate before calling

if autocommit_duration_ms is not None and autocommit_duration_ms <= 0:
    raise SystemExit("autocommit_duration_ms must be positive milliseconds or None")

t = pw.io.kafka.read(rdkafka_settings, topic="t", autocommit_duration_ms=autocommit_duration_ms)

Type guard

def valid_autocommit(v) -> bool:
    return v is None or (isinstance(v, int) and v > 0)

Prevention

Use a sane default like 10_000 ms; do not set 0 hoping to disable commits.
Remember this value bounds at-least-once re-read volume after crashes — larger means fewer commits but more potential duplicates.

When it happens

Trigger: pw.io.kafka.read(..., autocommit_duration_ms=0) or a negative value. None uses the connector default; only explicit non-positive integers raise.

Common situations: Setting autocommit_duration_ms=0 intending to 'disable autocommit' when the developer actually wants manual/None behavior; unit confusion (seconds vs ms) producing 0 after integer division.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/dba74844e4758c27. Report an issue: GitHub.