apache/beam · error · ValueError

num_splits must be greater than or equal 0

Error message

num_splits must be greater than or equal 0

What it means

ReadFromDatastore validates that num_splits is a non-negative integer; a negative value would request a nonsensical (impossible) number of query splits, so it is rejected before the split query is built.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:130

  def __init__(self, query, num_splits=0):
    """Initialize the `ReadFromDatastore` transform.

    This transform outputs elements of type
    :class:`~apache_beam.io.gcp.datastore.v1new.types.Entity`.

    Args:
      query: (:class:`~apache_beam.io.gcp.datastore.v1new.types.Query`) query
        used to fetch entities.
      num_splits: (:class:`int`) (optional) Number of splits for the query.
    """
    super().__init__()

    if not query.project:
      raise ValueError("query.project cannot be empty")
    if not query:
      raise ValueError("query cannot be empty")
    if num_splits < 0:
      raise ValueError("num_splits must be greater than or equal 0")

    self._project = query.project
    # using _namespace conflicts with DisplayData._namespace
    self._datastore_namespace = query.namespace
    self._query = query
    self._num_splits = num_splits

  def expand(self, pcoll):
    # This is a composite transform involves the following:
    #   1. Create a singleton of the user provided `query` and apply a ``ParDo``
    #   that splits the query into `num_splits` queries if possible.
    #
    #   If the value of `num_splits` is 0, the number of splits will be
    #   computed dynamically based on the size of the data for the `query`.
    #
    #   2. The resulting ``PCollection`` is sharded across workers using a
    #   ``Reshuffle`` operation.
    #

View on GitHub (pinned to 12126d8942)

Solutions

1. Clamp the value: num_splits = max(0, num_splits) before constructing the transform.
2. Pass num_splits=None (or omit) to let the connector pick a sensible number of splits.
3. Fix the computation/config that produced the negative split count.

Example fix

// before
ReadFromDatastore(query=q, num_splits=-1)  # ValueError

// after
ReadFromDatastore(query=q, num_splits=max(0, desired_splits))

if num_splits is not None and num_splits < 0:
    raise ValueError('num_splits must be >= 0')

def is_valid_num_splits(n):
    return n is None or (isinstance(n, int) and n >= 0)

try:
    read = ReadFromDatastore(query=query, num_splits=n)
except ValueError as e:
    if 'num_splits' in str(e):
        read = ReadFromDatastore(query=query)  # let library choose splits
    else:
        raise

When it happens

Trigger: Calling ReadFromDatastore(query=q, num_splits=-1) or any negative int, typically from a miscomputed config value or an integer parse that produced a negative number.

Common situations: Autoscaling code computing splits as (target - actual) which can go negative; user config typo '-1'; uninitialized default of -1 used as a sentinel.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• query.project cannot be empty
• query cannot be empty
• Datastore total statistics unavailable.
• Cannot specify both 'labels' and 'index'/'columns'
• num_splits must be > 1, got: %d

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/563359d8eb3d96ec. Report an issue: GitHub.