apache/beam · error · NoSuchElementException

Datastore statistics for kind {kind} unavailable

Error message

Datastore statistics for kind {kind} unavailable

What it means

getLatestTableStats queries Datastore's per-kind statistics entity (e.g. __Stat_Kind__-derived rows) for a specific kind and returns the first result. If the query succeeds but yields zero entity results, no statistics entry exists for that kind yet, and a NoSuchElementException is thrown. This typically happens because Datastore only generates kind statistics periodically for kinds that contain data.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/datastore/DatastoreV1.java:471

      } else {
        queryBuilder.addKindBuilder().setName("__Stat_Ns_Kind__");
      }

      queryBuilder.setFilter(
          makeAndFilter(
              makeFilter("kind_name", EQUAL, makeValue(ourKind).build()).build(),
              makeFilter("timestamp", EQUAL, makeValue(latestTimestamp).build()).build()));

      RunQueryRequest request =
          makeRequest(projectId, databaseId, queryBuilder.build(), namespace, readTime);

      long now = System.currentTimeMillis();
      RunQueryResponse response = datastore.runQuery(request);
      LOG.debug("Query for per-kind statistics took {}ms", System.currentTimeMillis() - now);

      QueryResultBatch batch = response.getBatch();
      if (batch.getEntityResultsCount() == 0) {
        throw new NoSuchElementException(
            "Datastore statistics for kind " + ourKind + " unavailable");
      }
      return batch.getEntityResults(0).getEntity();
    }

    /**
     * Get the estimated size of the data returned by the given query.
     *
     * <p>Cloud Datastore provides no way to get a good estimate of how large the result of a query
     * entity kind being queried, using the __Stat_Kind__ system table, assuming exactly 1 kind is
     * specified in the query.
     *
     * <p>See https://cloud.google.com/datastore/docs/concepts/stats.
     */
    static long getEstimatedSizeBytes(
        Datastore datastore,
        String projectId,
        String databaseId,

View on GitHub (pinned to 12126d8942)

Solutions

Ensure the kind has entities and wait up to 24 hours for Datastore statistics generation, then retry
Check that the namespace parameter matches the namespace containing the kind's entities
Catch NoSuchElementException around getLatestTableStats and skip/defer size-based split logic (treat stats as unavailable)
Retry the query later rather than immediately after first writes to the kind

Example fix

// before
Entity stats = getLatestTableStats(projectId, kind, namespace, null);
// after
Entity stats;
try {
  stats = getLatestTableStats(projectId, kind, namespace, null);
} catch (NoSuchElementException e) {
  LOG.warn("No Datastore stats for kind {}; proceeding without size estimates", kind);
  stats = null;
}

Defensive patterns

Strategy: try-catch

Validate before calling

// Verify the kind has entities before requesting stats:
Query q = Query.newGqlQueryBuilder(Query.ResultType.ENTITY,
    "SELECT * FROM __Stat_Kind__ WHERE __key__ HAS ANCESTOR KEY('__Stat_Kind__', '" + kind + "')")
    .setLimit(1).build();
boolean kindStatsExist = datastore.runQuery(q).getBatch().getEntityResultsCount() > 0;

Try / catch

Entity stats;
try {
  stats = getLatestTableStats(projectId, kind, namespace, readTime);
} catch (NoSuchElementException e) {
  LOG.warn("Datastore stats unavailable for kind {}", kind, e);
  stats = null;
}

Prevention

Ensure the kind contains entities before expecting statistics
Double-check namespace parameters against your Datastore console
Add a retry/backoff around statistics lookups for freshly written kinds

When it happens

Trigger: Calling the entity() sink path that resolves per-kind table stats for a kind whose __Stat_Kind__ query returns no rows — new/empty kind, wrong namespace, or statistics not yet baked at the requested readTime.

Common situations: Writing to a newly created Datastore kind before statistics exist; namespace mismatch (kind exists in another namespace); running soon after first writes within the ~24h statistics refresh window; readTime snapshot older than any statistics.

Understand the failure class

Background: "Not found" and "does not exist" errors: why "Task not found", "No such folder", and "Can't find" fire when a lookup comes back empty — this error's family across 14 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8f7c3f35b9359d7e. Report an issue: GitHub.