{"record":{"id":"902326c24b8470ad","repo":"apache/beam","slug":"user-query-might-have-a-limit-already-set-so-trying-without","errorCode":null,"errorMessage":"User query might have a limit already set, so trying without zero limit","messagePattern":"User query might have a limit already set, so trying without zero limit","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java","lineNumber":620,"sourceCode":"        String projectId,\n        String databaseId,\n        String namespace,\n        @Nullable Instant readTime)\n        throws DatastoreException {\n      String gqlQueryWithZeroLimit = gql + \" LIMIT 0\";\n      try {\n        Query translatedQuery =\n            translateGqlQuery(\n                gqlQueryWithZeroLimit, datastore, projectId, databaseId, namespace, readTime);\n        // Clear the limit that we set.\n        return translatedQuery.toBuilder().clearLimit().build();\n      } catch (DatastoreException e) {\n        // Note: There is no specific error code or message to detect if the query already has a\n        // limit, so we just check for INVALID_ARGUMENT and assume that that the query might have\n        // a limit already set.\n        if (e.getCode() == Code.INVALID_ARGUMENT) {\n          LOG.warn(\"Failed to translate Gql query '{}'\", gqlQueryWithZeroLimit, e);\n          LOG.warn(\"User query might have a limit already set, so trying without zero limit\");\n          // Retry without the zero limit.\n          return translateGqlQuery(gql, datastore, projectId, databaseId, namespace, readTime);\n        } else {\n          throw e;\n        }\n      }\n    }\n\n    /** Translates a gql query string to {@link Query}. */\n    private static Query translateGqlQuery(\n        String gql,\n        Datastore datastore,\n        String projectId,\n        String databaseId,\n        String namespace,\n        @Nullable Instant readTime)\n        throws DatastoreException {\n      GqlQuery gqlQuery = GqlQuery.newBuilder().setQueryString(gql).setAllowLiterals(true).build();","sourceCodeStart":602,"sourceCodeEnd":638,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java#L602-L638","documentation":"When translating a GQL string query into a Cloud Datastore protobuf Query, Beam first attempts the translation with a zero LIMIT so that splitting can work correctly. If the Cloud Datastore API rejects that translation with an INVALID_ARGUMENT error, the connector assumes the user's GQL string already contained its own LIMIT clause, logs this warning, and silently retries the translation without the zero limit. This is a heuristic workaround because the API has no dedicated error code distinguishing 'limit already set' from other invalid arguments.","triggerScenarios":"Running DatastoreIO.read via V1 newQuery(gql) with a GQL query string that includes its own 'LIMIT n' clause, causing the zero-limit rewrite to fail with code INVALID_ARGUMENT; also any GQL string that is structurally invalid but happens to be caught by this catch block.","commonSituations":"Users hand-writing GQL like 'SELECT * FROM Kind LIMIT 100' and passing it to DatastoreIO.v1().read(); users who copied a paginated query from the Datastore console into a Beam pipeline.","solutions":["Remove the LIMIT clause from your GQL string; use Read.withLimit(numResults) on the DatastoreIO.Read transform instead so the connector controls limits and can still split the query.","If a limit is truly required, accept this warning: it is benign and the connector retries the translation without the zero limit automatically.","If translation still fails, validate the GQL syntax independently (e.g. run it in the Datastore console) to rule out a genuinely invalid query."],"exampleFix":"// before\nDatastoreIO.v1().read().withQuery(\"SELECT * FROM Person LIMIT 100\")\n// after\nDatastoreIO.v1().read().withQuery(\"SELECT * FROM Person\").withLimit(100)","handlingStrategy":"validation","validationCode":"// Java: strip LIMIT from GQL before handing to DatastoreIO\njava.util.regex.Matcher m = java.util.regex.Pattern\n    .compile(\"(?i)\\\\s+limit\\\\s+\\\\d+\\\\s*$\").matcher(gql);\nif (m.find()) {\n  gql = gql.substring(0, m.start());\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Never put LIMIT in GQL strings for DatastoreIO; use withLimit() on the Read transform.","Treat this warning as informational when a limit in GQL is intentional."],"tags":["gcp-datastore","gql","query-limit","beam-io"],"backgroundTag":"invalid-argument-value","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}