What causes "Query cannot have a limit set." in apache/beam?

Set query.limit = None (or don't set it) before requesting splits.

apache/beam · error · SplitNotPossibleError

Query cannot have a limit set.

Error message

Query cannot have a limit set.

What it means

validate_split rejects queries with a limit, because a limit only makes sense over the whole result set; after splitting, each shard would apply its own limit, changing semantics. SplitNotPossibleError is raised when query.limit is not None.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py:101

  return splits


def validate_split(query):
  """
  Verifies that the given query can be properly scattered.

  Note that equality and ancestor filters are allowed, however they may result
  in inefficient sharding.

  Raises:
    QuerySplitterError if split could not be performed owing to query
      parameters.
  """
  if query.order:
    raise SplitNotPossibleError('Query cannot have any sort orders.')

  if query.limit is not None:
    raise SplitNotPossibleError('Query cannot have a limit set.')

  for filter in query.filters:
    if isinstance(filter[1], ValueProvider):
      filter_operator = filter[1].get()
    else:
      filter_operator = filter[1]
    if filter_operator in ['<', '<=', '>', '>=']:
      raise SplitNotPossibleError('Query cannot have any inequality filters.')


def _create_scatter_query(query, num_splits):
  """Creates a scatter query from the given user query."""
  # There is a split containing entities before and after each scatter entity:
  # ||---*------*------*------*------*------*------*---||  * = scatter entity
  # If we represent each split as a region before a scatter entity, there is an
  # extra region following the last scatter point. Thus, we do not need the
  # scatter entity for the last region.
  limit = (num_splits - 1) * KEYS_PER_SPLIT

View on GitHub (pinned to 12126d8942)

Solutions

Set query.limit = None (or don't set it) before requesting splits.
Create a limit-free copy of the query for splitting and apply the limit afterwards/externally.
Catch SplitNotPossibleError and fall back to a single unsplit query that honors the limit.
If you only need N rows, don't split — run the limited query directly.

Example fix

// before
query.limit = 1000
splits = query_splitter.get_splits(client, query, n)
// after
query.limit = None
splits = query_splitter.get_splits(client, query, n)

Defensive patterns

Strategy: validation

Validate before calling

if query.limit is not None:
    raise ValueError('clear query.limit before splitting')
splits = query_splitter.get_splits(client, query, n)

Type guard

def is_splittable_query(query):
    return not query.order and query.limit is None

Try / catch

try:
    splits = query_splitter.get_splits(client, query, n)
except SplitNotPossibleError:
    logging.warning('query has a limit; running unsplit')
    splits = [query]

Prevention

Never set limit on queries destined for the splitter
Avoid reusing preview/debug queries (with limits) in production splits
If you need capped work, cap workers, not query limit
Run validate_split early to fail fast with a clear message

When it happens

Trigger: Calling get_splits/validate_split with query.limit set (e.g. query.limit = 100) — commonly to preview data — then attempting parallel split execution.

Common situations: Reusing a debugging/preview query (with limit) in the splitting pipeline; ORM-ish query builders that default a limit; capping cost of reads then trying to parallelize them.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/be9e1f6b1d918cf1. Report an issue: GitHub.