apache/beam · error · RuntimeError

Datastore total statistics unavailable.

Error message

Datastore total statistics unavailable.

What it means

query_latest_statistics_timestamp() queries the Datastore __Stat_Total__ statistics kind for the latest timestamp entry and raises RuntimeError when the fetch returns no entities. This happens when Datastore has not yet computed overall statistics for the project (new/empty projects or recently written data).

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:223

    @staticmethod
    def query_latest_statistics_timestamp(client):
      """Fetches the latest timestamp of statistics from Cloud Datastore.

      Cloud Datastore system tables with statistics are periodically updated.
      This method fetches the latest timestamp (in microseconds) of statistics
      update using the `__Stat_Total__` table.
      """
      if client.namespace is None:
        kind = '__Stat_Total__'
      else:
        kind = '__Stat_Ns_Total__'
      query = client.query(
          kind=kind, order=[
              "-timestamp",
          ])
      entities = list(query.fetch(limit=1))
      if not entities:
        raise RuntimeError("Datastore total statistics unavailable.")
      return entities[0]['timestamp']

    @staticmethod
    def get_estimated_size_bytes(client, query):
      """Get the estimated size of the data returned by this instance's query.

      Cloud Datastore provides no way to get a good estimate of how large the
      result of a query is going to be. Hence we use the __Stat_Kind__ system
      table to get size of the entire kind as an approximate estimate, assuming
      exactly 1 kind is specified in the query.
      See https://cloud.google.com/datastore/docs/concepts/stats.
      """
      kind_name = query.kind
      latest_timestamp = (
          ReadFromDatastore._SplitQueryFn.query_latest_statistics_timestamp(
              client))
      _LOGGER.info(
          'Latest stats timestamp for kind %s is %s',

View on GitHub (pinned to 12126d8942)

Solutions

1. Provide an explicit estimated size instead of relying on statistics-based splitting (bypass query_latest_statistics_timestamp).
2. Wait for Datastore statistics to populate (stats update periodically) and retry.
3. Verify you are querying the correct project/database where __Stat_Total__ entries exist; for Firestore-in-Datastore-mode namespaces, query namespace-specific stats.
4. Catch the RuntimeError and fall back to a default split count.

Example fix

// before
timestamp = query_latest_statistics_timestamp(client, kind)  # RuntimeError if no stats

// after
try:
    timestamp = query_latest_statistics_timestamp(client, kind)
except RuntimeError:
    timestamp = None  # fall back to default splitting

from google.cloud import datastore
client = datastore.Client(project=project)
q = client.query(kind='__Stat_Total__')
if not list(q.fetch(limit=1)):
    # stats not yet available; plan for a fallback
    pass

None

try:
    ts = query_latest_statistics_timestamp(client, kind)
except RuntimeError as e:
    if 'statistics unavailable' in str(e):
        ts = None  # fall back to fixed split count / default size estimate
    else:
        raise

When it happens

Trigger: Calling Read/Write with size estimation enabled against a project whose __Stat_Total__ kind has no rows (empty project, statistics not yet computed, or a database/mode where total stats are unavailable).

Common situations: Brand-new GCP projects with no statistics yet; Datastore mode Firestore projects where total stats lag or are absent; running size-estimation-based splitting immediately after bulk writes before stats refresh.

Understand the failure class

Background: EmptyResultError / "no results found": when an API or scraper succeeds but returns zero rows — this error's family across 9 libraries.

Related errors

• query.project cannot be empty
• query cannot be empty
• num_splits must be greater than or equal 0
• Datastore total statistics unavailable
• Datastore statistics for kind {kind} unavailable

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e54188e35938fc2f. Report an issue: GitHub.