What causes "num_splits must be > 1, got: %d" in apache/beam?

Ensure num_splits >= 2 before calling get_splits (e.g. max(2, desired_splits)).

apache/beam · error · SplitNotPossibleError

num_splits must be > 1, got: %d

Error message

num_splits must be > 1, got: %d

What it means

Datastore query splitting (get_splits) requires num_splits > 1 because splitting produces boundaries between at least two regions; asking for 1 (or fewer, or negative) splits is meaningless, so SplitNotPossibleError (a QuerySplitterError) is raised before any scatter-query work.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py:72

  gather random split points for a query.

  Note: This implementation is derived from the java query splitter in
  https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/master/java/datastore/src/main/java/com/google/datastore/v1/client/QuerySplitterImpl.java

  Args:
    client: the datastore client.
    query: the query to split.
    num_splits: the desired number of splits.

  Returns:
    A list of split queries, of a max length of `num_splits`

  Raises:
    QuerySplitterError: if split could not be performed owing to query or split
      parameters.
  """
  if num_splits <= 1:
    raise SplitNotPossibleError('num_splits must be > 1, got: %d' % num_splits)
  validate_split(query)

  splits = []
  client_scatter_keys = _get_scatter_keys(client, query, num_splits)
  last_client_key = None
  for next_client_key in _get_split_key(client_scatter_keys, num_splits):
    splits.append(_create_split(last_client_key, next_client_key, query))
    last_client_key = next_client_key

  splits.append(_create_split(last_client_key, None, query))
  return splits


def validate_split(query):
  """
  Verifies that the given query can be properly scattered.

  Note that equality and ancestor filters are allowed, however they may result

View on GitHub (pinned to 12126d8942)

Solutions

Ensure num_splits >= 2 before calling get_splits (e.g. max(2, desired_splits)).
Skip splitting entirely when only one split is needed — query without a split.
Fix the computation producing num_splits (check for zero remaining jobs/workers) and clamp it.
Catch SplitNotPossibleError and fall back to an unsplit query.

Example fix

// before
splits = query_splitter.get_splits(client, query, num_splits)  # num_splits may be 1
// after
if num_splits > 1:
    splits = query_splitter.get_splits(client, query, num_splits)
else:
    splits = [query]

Defensive patterns

Strategy: validation

Validate before calling

if num_splits is None or num_splits <= 1:
    raise ValueError('num_splits must be >= 2 before calling get_splits')

Type guard

def can_split(num_splits):
    return isinstance(num_splits, int) and num_splits > 1

Try / catch

try:
    splits = query_splitter.get_splits(client, query, num_splits)
except SplitNotPossibleError:
    splits = [query]  # run unsplit

Prevention

Clamp computed split counts: num_splits = max(2, computed)
Guard dynamic work rebalancing against zero/negative remaining work
Beware integer division producing 0 or 1 for small worker counts
Only call the splitter when parallelism > 1

When it happens

Trigger: Calling get_splits(client, query, num_splits) with num_splits <= 1 — e.g. passing num_splits=1, 0, or a computed value like int(total_workers / num_jobs) that evaluates to 1 or 0 when few workers/jobs are configured.

Common situations: Dynamic work rebalancing computing splits from remaining work counts that reach 0; configured parallelism of 1; integer division rounding down to 1 or 0 for small inputs.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9438330e8f6f85bb. Report an issue: GitHub.