apache/beam · warning

User query might have a limit already set, so trying without

Error message

User query might have a limit already set, so trying without zero limit

What it means

When translating a GQL string query into a Cloud Datastore protobuf Query, Beam first attempts the translation with a zero LIMIT so that splitting can work correctly. If the Cloud Datastore API rejects that translation with an INVALID_ARGUMENT error, the connector assumes the user's GQL string already contained its own LIMIT clause, logs this warning, and silently retries the translation without the zero limit. This is a heuristic workaround because the API has no dedicated error code distinguishing 'limit already set' from other invalid arguments.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java:620

        String projectId,
        String databaseId,
        String namespace,
        @Nullable Instant readTime)
        throws DatastoreException {
      String gqlQueryWithZeroLimit = gql + " LIMIT 0";
      try {
        Query translatedQuery =
            translateGqlQuery(
                gqlQueryWithZeroLimit, datastore, projectId, databaseId, namespace, readTime);
        // Clear the limit that we set.
        return translatedQuery.toBuilder().clearLimit().build();
      } catch (DatastoreException e) {
        // Note: There is no specific error code or message to detect if the query already has a
        // limit, so we just check for INVALID_ARGUMENT and assume that that the query might have
        // a limit already set.
        if (e.getCode() == Code.INVALID_ARGUMENT) {
          LOG.warn("Failed to translate Gql query '{}'", gqlQueryWithZeroLimit, e);
          LOG.warn("User query might have a limit already set, so trying without zero limit");
          // Retry without the zero limit.
          return translateGqlQuery(gql, datastore, projectId, databaseId, namespace, readTime);
        } else {
          throw e;
        }
      }
    }

    /** Translates a gql query string to {@link Query}. */
    private static Query translateGqlQuery(
        String gql,
        Datastore datastore,
        String projectId,
        String databaseId,
        String namespace,
        @Nullable Instant readTime)
        throws DatastoreException {
      GqlQuery gqlQuery = GqlQuery.newBuilder().setQueryString(gql).setAllowLiterals(true).build();

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove the LIMIT clause from your GQL string; use Read.withLimit(numResults) on the DatastoreIO.Read transform instead so the connector controls limits and can still split the query.
2. If a limit is truly required, accept this warning: it is benign and the connector retries the translation without the zero limit automatically.
3. If translation still fails, validate the GQL syntax independently (e.g. run it in the Datastore console) to rule out a genuinely invalid query.

Example fix

// before
DatastoreIO.v1().read().withQuery("SELECT * FROM Person LIMIT 100")
// after
DatastoreIO.v1().read().withQuery("SELECT * FROM Person").withLimit(100)

// Java: strip LIMIT from GQL before handing to DatastoreIO
java.util.regex.Matcher m = java.util.regex.Pattern
    .compile("(?i)\\s+limit\\s+\\d+\\s*$").matcher(gql);
if (m.find()) {
  gql = gql.substring(0, m.start());
}

When it happens

Trigger: Running DatastoreIO.read via V1 newQuery(gql) with a GQL query string that includes its own 'LIMIT n' clause, causing the zero-limit rewrite to fail with code INVALID_ARGUMENT; also any GQL string that is structurally invalid but happens to be caught by this catch block.

Common situations: Users hand-writing GQL like 'SELECT * FROM Kind LIMIT 100' and passing it to DatastoreIO.v1().read(); users who copied a paginated query from the Datastore console into a Beam pipeline.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

• Unable to parallelize the given query: {}
• Mutation {} does not have an operation type set.
• overloadRatio must be greater than 1.0
• throttleDelaySecs must be greater than 0
• requires an input Schema. Note that only Row or user classe

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/902326c24b8470ad. Report an issue: GitHub.