apache/beam · error · RuntimeError

Datastore statistics for kind %s unavailable

Error message

Datastore statistics for kind %s unavailable

What it means

Raised by ReadFromDatastore's helper when querying Cloud Datastore's built-in __stat__ entities: no statistics entity exists yet for the requested kind, so an estimated size cannot be returned. Datastore only refreshes kind statistics periodically (up to ~24-30h), so new or rarely-used kinds have no stats row.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:255

      latest_timestamp = (
          ReadFromDatastore._SplitQueryFn.query_latest_statistics_timestamp(
              client))
      _LOGGER.info(
          'Latest stats timestamp for kind %s is %s',
          kind_name,
          latest_timestamp)

      if client.namespace is None:
        kind = '__Stat_Kind__'
      else:
        kind = '__Stat_Ns_Kind__'
      query = client.query(kind=kind)
      query.add_filter('kind_name', '=', kind_name)
      query.add_filter('timestamp', '=', latest_timestamp)

      entities = list(query.fetch(limit=1))
      if not entities:
        raise RuntimeError(
            'Datastore statistics for kind %s unavailable' % kind_name)
      return entities[0]['entity_bytes']

    @staticmethod
    def get_estimated_num_splits(client, query):
      """Computes the number of splits to be performed on the query."""
      try:
        estimated_size_bytes = (
            ReadFromDatastore._SplitQueryFn.get_estimated_size_bytes(
                client, query))
        _LOGGER.info('Estimated size bytes for query: %s', estimated_size_bytes)
        num_splits = int(
            min(
                ReadFromDatastore._NUM_QUERY_SPLITS_MAX,
                round((
                    float(estimated_size_bytes) /
                    ReadFromDatastore._DEFAULT_BUNDLE_SIZE_BYTES))))
      except Exception as e:

View on GitHub (pinned to 12126d8942)

Solutions

  1. Wait for Datastore kind statistics to refresh (stats update roughly daily) and retry the read.
  2. Provide an explicit hint_num_workers or avoid size-based split estimation so statistics are not consulted.
  3. Verify the kind name spelling and that the kind actually contains entities in the target project/namespace.
  4. Catch RuntimeError and fall back to a default estimated size.
  5. Seed the kind with data and/or run a query so Datastore generates the __stat__ entry before estimating.

Example fix

// before
estimated = ReadFromDatastore(project, query, hint_num_workers=0)  # triggers stats lookup
// after
try:
    size = helper.get_estimated_size_bytes(client, kind)
except RuntimeError:
    size = DEFAULT_FALLBACK_SIZE_BYTES
Defensive patterns

Strategy: fallback

Validate before calling

stat_entities = list(client.query(kind='__stat__').add_filter('kind_name', '=', kind_name).fetch(limit=1))
if not stat_entities:
    logging.warning('No Datastore statistics yet for kind %s; using fallback size', kind_name)

Type guard

def has_stats(entities):
    return bool(entities) and 'entity_bytes' in entities[0]

Try / catch

try:
    size = get_estimated_size_bytes(client, kind)
except RuntimeError:
    size = DEFAULT_ESTIMATED_SIZE_BYTES

Prevention

When it happens

Trigger: Calling get_estimated_size_bytes (used by ReadFromDatastore with hint_num_workers) for a kind whose __stat__ entity (filtered by kind_name and latest timestamp) has not been created yet — e.g. the kind has never been written, was just created, or stats have not refreshed since the last write.

Common situations: Pipelines that write a kind and immediately read it back with size-based splitting; brand-new kinds in fresh projects; kinds with zero entities (some stat rows appear only after traffic); running estimate shortly after bulk delete/insert before the nightly stats job runs.

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/385e47bf5e313c70. Report an issue: GitHub.