{"record":{"id":"e54188e35938fc2f","repo":"apache/beam","slug":"datastore-total-statistics-unavailable-datastoreio","errorCode":null,"errorMessage":"Datastore total statistics unavailable.","messagePattern":"Datastore total statistics unavailable\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py","lineNumber":223,"sourceCode":"    @staticmethod\n    def query_latest_statistics_timestamp(client):\n      \"\"\"Fetches the latest timestamp of statistics from Cloud Datastore.\n\n      Cloud Datastore system tables with statistics are periodically updated.\n      This method fetches the latest timestamp (in microseconds) of statistics\n      update using the `__Stat_Total__` table.\n      \"\"\"\n      if client.namespace is None:\n        kind = '__Stat_Total__'\n      else:\n        kind = '__Stat_Ns_Total__'\n      query = client.query(\n          kind=kind, order=[\n              \"-timestamp\",\n          ])\n      entities = list(query.fetch(limit=1))\n      if not entities:\n        raise RuntimeError(\"Datastore total statistics unavailable.\")\n      return entities[0]['timestamp']\n\n    @staticmethod\n    def get_estimated_size_bytes(client, query):\n      \"\"\"Get the estimated size of the data returned by this instance's query.\n\n      Cloud Datastore provides no way to get a good estimate of how large the\n      result of a query is going to be. Hence we use the __Stat_Kind__ system\n      table to get size of the entire kind as an approximate estimate, assuming\n      exactly 1 kind is specified in the query.\n      See https://cloud.google.com/datastore/docs/concepts/stats.\n      \"\"\"\n      kind_name = query.kind\n      latest_timestamp = (\n          ReadFromDatastore._SplitQueryFn.query_latest_statistics_timestamp(\n              client))\n      _LOGGER.info(\n          'Latest stats timestamp for kind %s is %s',","sourceCodeStart":205,"sourceCodeEnd":241,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py#L205-L241","documentation":"query_latest_statistics_timestamp() queries the Datastore __Stat_Total__ statistics kind for the latest timestamp entry and raises RuntimeError when the fetch returns no entities. This happens when Datastore has not yet computed overall statistics for the project (new/empty projects or recently written data).","triggerScenarios":"Calling Read/Write with size estimation enabled against a project whose __Stat_Total__ kind has no rows (empty project, statistics not yet computed, or a database/mode where total stats are unavailable).","commonSituations":"Brand-new GCP projects with no statistics yet; Datastore mode Firestore projects where total stats lag or are absent; running size-estimation-based splitting immediately after bulk writes before stats refresh.","solutions":["Provide an explicit estimated size instead of relying on statistics-based splitting (bypass query_latest_statistics_timestamp).","Wait for Datastore statistics to populate (stats update periodically) and retry.","Verify you are querying the correct project/database where __Stat_Total__ entries exist; for Firestore-in-Datastore-mode namespaces, query namespace-specific stats.","Catch the RuntimeError and fall back to a default split count."],"exampleFix":"// before\ntimestamp = query_latest_statistics_timestamp(client, kind)  # RuntimeError if no stats\n\n// after\ntry:\n    timestamp = query_latest_statistics_timestamp(client, kind)\nexcept RuntimeError:\n    timestamp = None  # fall back to default splitting","handlingStrategy":"fallback","validationCode":"from google.cloud import datastore\nclient = datastore.Client(project=project)\nq = client.query(kind='__Stat_Total__')\nif not list(q.fetch(limit=1)):\n    # stats not yet available; plan for a fallback\n    pass","typeGuard":"None","tryCatchPattern":"try:\n    ts = query_latest_statistics_timestamp(client, kind)\nexcept RuntimeError as e:\n    if 'statistics unavailable' in str(e):\n        ts = None  # fall back to fixed split count / default size estimate\n    else:\n        raise","preventionTips":["For new or nearly empty projects, supply an explicit size estimate instead of relying on __Stat_Total__.","Remember Datastore statistics refresh periodically; avoid splitting immediately after bulk loads.","For Firestore-in-Datastore-mode, check whether total statistics exist for your database/namespace.","Always wrap statistics lookups with a fallback path in pipeline construction."],"tags":["python","apache-beam","datastore","gcp","statistics"],"backgroundTag":"empty-result-set","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}