pathwaycom/pathway · error · ValueError

'read_only_new=True' and 'start_from_timestamp_ms' both cont

Error message

'read_only_new=True' and 'start_from_timestamp_ms' both control the starting position and conflict. Pick one — drop 'read_only_new=True' if you want to start from the timestamp, or drop 'start_from_timestamp_ms' if you want to start from the end.

What it means

In pw.io.kafka.simple_read, read_only_new=True (seek to end) and start_from_timestamp_ms (seek to a time offset) are two different ways of choosing the consumer's starting position and cannot both apply. Pathway raises ValueError asking you to pick one.

Source

Thrown at python/pathway/io/kafka/__init__.py:510

        if forbidden in kwargs:
            raise TypeError(
                f"'simple_read' does not accept {forbidden!r}. "
                "If you need to customize the rdkafka settings (auth, TLS, "
                "consumer group, etc.) use 'pw.io.kafka.read' directly; "
                "'simple_read' is a thin wrapper that builds these settings "
                "itself."
            )

    if read_only_new and kwargs.get("mode") == "static":
        raise ValueError(
            "'read_only_new=True' together with mode='static' is "
            "self-contradictory: 'read_only_new' starts the consumer at the "
            "end of the partition, so the static reader has nothing to "
            "consume. Pick one — drop 'read_only_new=True' or use "
            "'mode=\"streaming\"' (the default)."
        )
    if read_only_new and kwargs.get("start_from_timestamp_ms") is not None:
        raise ValueError(
            "'read_only_new=True' and 'start_from_timestamp_ms' both control "
            "the starting position and conflict. Pick one — drop "
            "'read_only_new=True' if you want to start from the timestamp, "
            "or drop 'start_from_timestamp_ms' if you want to start from "
            "the end."
        )

    rdkafka_settings = {
        "bootstrap.servers": server,
        "group.id": str(uuid.uuid4()),
        "auto.offset.reset": "end" if read_only_new else "beginning",
    }
    return read(
        rdkafka_settings=rdkafka_settings,
        topic=topic,
        schema=schema,
        format=format,
        debug_data=debug_data,

View on GitHub (pinned to fa2f74a464)

Solutions

Keep start_from_timestamp_ms and drop read_only_new if you want to start at a specific time.
Keep read_only_new=True and drop start_from_timestamp_ms if you want the very end of the partition.
For full control (both concerns, custom offsets), use pw.io.kafka.read with explicit rdkafka_settings.

Example fix

# before
t = pw.io.kafka.simple_read(
    "localhost:9092", "test-topic",
    read_only_new=True, start_from_timestamp_ms=1700000000000,
)
# after
t = pw.io.kafka.simple_read(
    "localhost:9092", "test-topic",
    start_from_timestamp_ms=1700000000000,
)

Defensive patterns

Strategy: validation

Validate before calling

if read_only_new and start_from_timestamp_ms is not None:
    raise SystemExit(
        "Pick one starting position: read_only_new=True OR start_from_timestamp_ms"
    )

t = pw.io.kafka.simple_read(
    "localhost:9092", "test-topic",
    read_only_new=read_only_new,
    start_from_timestamp_ms=start_from_timestamp_ms,
)

Type guard

def compatible_start_opts(read_only_new: bool, ts) -> bool:
    return not (read_only_new and ts is not None)

Prevention

Model 'start position' as a single config field (none|end|timestamp) instead of independent booleans.
Validate the combined option set once at startup.

When it happens

Trigger: pw.io.kafka.simple_read('localhost:9092', 'test-topic', read_only_new=True, start_from_timestamp_ms=1700000000000). Internally simple_read sets auto.offset.reset to 'end' when read_only_new is set, which would conflict with the timestamp seek.

Common situations: Stacking 'start from now-ish' flags while tuning startup behavior; forwarding both options from a shared config dict into simple_read.

Related errors

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/2a5fa19352fcaff6. Report an issue: GitHub.