{"record":{"id":"385e47bf5e313c70","repo":"apache/beam","slug":"datastore-statistics-for-kind-s-unavailable","errorCode":null,"errorMessage":"Datastore statistics for kind %s unavailable","messagePattern":"Datastore statistics for kind (.+?) unavailable","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py","lineNumber":255,"sourceCode":"      latest_timestamp = (\n          ReadFromDatastore._SplitQueryFn.query_latest_statistics_timestamp(\n              client))\n      _LOGGER.info(\n          'Latest stats timestamp for kind %s is %s',\n          kind_name,\n          latest_timestamp)\n\n      if client.namespace is None:\n        kind = '__Stat_Kind__'\n      else:\n        kind = '__Stat_Ns_Kind__'\n      query = client.query(kind=kind)\n      query.add_filter('kind_name', '=', kind_name)\n      query.add_filter('timestamp', '=', latest_timestamp)\n\n      entities = list(query.fetch(limit=1))\n      if not entities:\n        raise RuntimeError(\n            'Datastore statistics for kind %s unavailable' % kind_name)\n      return entities[0]['entity_bytes']\n\n    @staticmethod\n    def get_estimated_num_splits(client, query):\n      \"\"\"Computes the number of splits to be performed on the query.\"\"\"\n      try:\n        estimated_size_bytes = (\n            ReadFromDatastore._SplitQueryFn.get_estimated_size_bytes(\n                client, query))\n        _LOGGER.info('Estimated size bytes for query: %s', estimated_size_bytes)\n        num_splits = int(\n            min(\n                ReadFromDatastore._NUM_QUERY_SPLITS_MAX,\n                round((\n                    float(estimated_size_bytes) /\n                    ReadFromDatastore._DEFAULT_BUNDLE_SIZE_BYTES))))\n      except Exception as e:","sourceCodeStart":237,"sourceCodeEnd":273,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py#L237-L273","documentation":"Raised by ReadFromDatastore's helper when querying Cloud Datastore's built-in __stat__ entities: no statistics entity exists yet for the requested kind, so an estimated size cannot be returned. Datastore only refreshes kind statistics periodically (up to ~24-30h), so new or rarely-used kinds have no stats row.","triggerScenarios":"Calling get_estimated_size_bytes (used by ReadFromDatastore with hint_num_workers) for a kind whose __stat__ entity (filtered by kind_name and latest timestamp) has not been created yet — e.g. the kind has never been written, was just created, or stats have not refreshed since the last write.","commonSituations":"Pipelines that write a kind and immediately read it back with size-based splitting; brand-new kinds in fresh projects; kinds with zero entities (some stat rows appear only after traffic); running estimate shortly after bulk delete/insert before the nightly stats job runs.","solutions":["Wait for Datastore kind statistics to refresh (stats update roughly daily) and retry the read.","Provide an explicit hint_num_workers or avoid size-based split estimation so statistics are not consulted.","Verify the kind name spelling and that the kind actually contains entities in the target project/namespace.","Catch RuntimeError and fall back to a default estimated size.","Seed the kind with data and/or run a query so Datastore generates the __stat__ entry before estimating."],"exampleFix":"// before\nestimated = ReadFromDatastore(project, query, hint_num_workers=0)  # triggers stats lookup\n// after\ntry:\n    size = helper.get_estimated_size_bytes(client, kind)\nexcept RuntimeError:\n    size = DEFAULT_FALLBACK_SIZE_BYTES","handlingStrategy":"fallback","validationCode":"stat_entities = list(client.query(kind='__stat__').add_filter('kind_name', '=', kind_name).fetch(limit=1))\nif not stat_entities:\n    logging.warning('No Datastore statistics yet for kind %s; using fallback size', kind_name)","typeGuard":"def has_stats(entities):\n    return bool(entities) and 'entity_bytes' in entities[0]","tryCatchPattern":"try:\n    size = get_estimated_size_bytes(client, kind)\nexcept RuntimeError:\n    size = DEFAULT_ESTIMATED_SIZE_BYTES","preventionTips":["Avoid estimating sizes immediately after creating/populating a kind","Prefer explicit worker hints over stats-based splitting for fresh datasets","Confirm kind spelling and namespace before querying __stat__","Wrap stats lookups in a helper with a sensible default size"],"tags":["gcp","datastore","statistics","beam-io"],"backgroundTag":"resource-not-found","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}