{"record":{"id":"be9e1f6b1d918cf1","repo":"apache/beam","slug":"query-cannot-have-a-limit-set","errorCode":null,"errorMessage":"Query cannot have a limit set.","messagePattern":"Query cannot have a limit set\\.","errorType":"exception","errorClass":"SplitNotPossibleError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py","lineNumber":101,"sourceCode":"  return splits\n\n\ndef validate_split(query):\n  \"\"\"\n  Verifies that the given query can be properly scattered.\n\n  Note that equality and ancestor filters are allowed, however they may result\n  in inefficient sharding.\n\n  Raises:\n    QuerySplitterError if split could not be performed owing to query\n      parameters.\n  \"\"\"\n  if query.order:\n    raise SplitNotPossibleError('Query cannot have any sort orders.')\n\n  if query.limit is not None:\n    raise SplitNotPossibleError('Query cannot have a limit set.')\n\n  for filter in query.filters:\n    if isinstance(filter[1], ValueProvider):\n      filter_operator = filter[1].get()\n    else:\n      filter_operator = filter[1]\n    if filter_operator in ['<', '<=', '>', '>=']:\n      raise SplitNotPossibleError('Query cannot have any inequality filters.')\n\n\ndef _create_scatter_query(query, num_splits):\n  \"\"\"Creates a scatter query from the given user query.\"\"\"\n  # There is a split containing entities before and after each scatter entity:\n  # ||---*------*------*------*------*------*------*---||  * = scatter entity\n  # If we represent each split as a region before a scatter entity, there is an\n  # extra region following the last scatter point. Thus, we do not need the\n  # scatter entity for the last region.\n  limit = (num_splits - 1) * KEYS_PER_SPLIT","sourceCodeStart":83,"sourceCodeEnd":119,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/datastore/v1new/query_splitter.py#L83-L119","documentation":"validate_split rejects queries with a limit, because a limit only makes sense over the whole result set; after splitting, each shard would apply its own limit, changing semantics. SplitNotPossibleError is raised when query.limit is not None.","triggerScenarios":"Calling get_splits/validate_split with query.limit set (e.g. query.limit = 100) — commonly to preview data — then attempting parallel split execution.","commonSituations":"Reusing a debugging/preview query (with limit) in the splitting pipeline; ORM-ish query builders that default a limit; capping cost of reads then trying to parallelize them.","solutions":["Set query.limit = None (or don't set it) before requesting splits.","Create a limit-free copy of the query for splitting and apply the limit afterwards/externally.","Catch SplitNotPossibleError and fall back to a single unsplit query that honors the limit.","If you only need N rows, don't split — run the limited query directly."],"exampleFix":"// before\nquery.limit = 1000\nsplits = query_splitter.get_splits(client, query, n)\n// after\nquery.limit = None\nsplits = query_splitter.get_splits(client, query, n)","handlingStrategy":"validation","validationCode":"if query.limit is not None:\n    raise ValueError('clear query.limit before splitting')\nsplits = query_splitter.get_splits(client, query, n)","typeGuard":"def is_splittable_query(query):\n    return not query.order and query.limit is None","tryCatchPattern":"try:\n    splits = query_splitter.get_splits(client, query, n)\nexcept SplitNotPossibleError:\n    logging.warning('query has a limit; running unsplit')\n    splits = [query]","preventionTips":["Never set limit on queries destined for the splitter","Avoid reusing preview/debug queries (with limits) in production splits","If you need capped work, cap workers, not query limit","Run validate_split early to fail fast with a clear message"],"tags":["python","datastore","query-validation","beam-io"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}