pathwaycom/pathway · error · ValueError

read_batch_size must be positive

Error message

read_batch_size must be positive

What it means

Raised by pw.io.elasticsearch.read when read_batch_size is zero or negative. The batch size controls how many documents are fetched per scroll request and must be a positive integer.

Source

Thrown at python/pathway/io/elasticsearch/__init__.py:330

        raise ValueError(
            f"timestamp_column {timestamp_column!r} is not present in the schema"
        )
    if id_column not in column_names:
        raise ValueError(f"id_column {id_column!r} is not present in the schema")
    if schema.primary_key_columns():
        raise ValueError(
            "Defining a primary key in the schema is not supported for "
            "pw.io.elasticsearch.read. The connector keys the resulting table by "
            "id_column. If you need to reindex the table by a different column, use "
            "pw.Table.with_id_from() after reading."
        )

    max_transaction_duration_ms = round(
        as_duration_seconds(max_transaction_duration, "max_transaction_duration") * 1000
    )
    poll_interval_ms = round(as_duration_seconds(poll_interval, "poll_interval") * 1000)
    if read_batch_size <= 0:
        raise ValueError("read_batch_size must be positive")

    data_storage = api.DataStorage(
        storage_type="elasticsearch",
        mode=internal_connector_mode(mode),
        elasticsearch_params=api.ElasticSearchParams(
            host=host,
            index_name=index_name,
            auth=auth.engine_es_auth,
        ),
        elasticsearch_reader_params=api.ElasticSearchReaderParams(
            timestamp_field=timestamp_column,
            id_field=id_column,
            max_transaction_duration_ms=max_transaction_duration_ms,
            read_batch_size=read_batch_size,
            poll_interval_ms=poll_interval_ms,
        ),
    )

View on GitHub (pinned to fa2f74a464)

Solutions

1. Pass a positive integer, e.g. read_batch_size=1000 (or simply omit the argument to use the default).
2. Validate/normalize config before the call: max(1, int(config.get("read_batch_size", 1000))).

Example fix

# before
pw.io.elasticsearch.read(..., read_batch_size=int(os.environ.get("BATCH", 0)))

# after
pw.io.elasticsearch.read(..., read_batch_size=max(1, int(os.environ.get("BATCH", 1000))))

read_batch_size = int(read_batch_size)
if read_batch_size <= 0:
    raise ValueError("read_batch_size must be a positive integer")

When it happens

Trigger: pw.io.elasticsearch.read(..., read_batch_size=0) or read_batch_size=-10, often via a config variable that defaulted to 0 or was computed from an expression that yielded a non-positive value.

Common situations: read_batch_size loaded from an environment variable or config file that is unset and defaults to 0; batch-size auto-tuning code that floors to 0 on small inputs.

Related errors

• parameters `schema` and `id_from` are mutually exclusive
• 'json_field_paths' references field {field_name!r} which is
• Invalid JSON Pointer for field {field_name!r}: {path!r}. JSO
• database {path_str!r} is an existing directory, not a DuckDB
• timestamp_column {timestamp_column!r} is not present in the

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/10af23feb18b5e3a. Report an issue: GitHub.