apache/beam · error · ValueError

apache_beam.io.gcp.datastore.v1new.datastoreio.Key expected,

Error message

apache_beam.io.gcp.datastore.v1new.datastoreio.Key expected, got: %s

What it means

DeleteFromDatastore's _DatastoreDeleteFn only accepts apache_beam.io.gcp.datastore.v1new.types.Key elements; any other element type raises ValueError identifying the actual received type. The check runs before converting to a client key.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:591

      project,
      throttle_rampup=True,
      hint_num_workers=_Mutate._DEFAULT_HINT_NUM_WORKERS):
    """Initialize the `DeleteFromDatastore` transform.

    Args:
      project: (:class:`str`) The ID of the project from which the entities will
        be deleted.
      throttle_rampup: Whether to enforce a gradual ramp-up.
      hint_num_workers: A hint for the expected number of workers, used to
                        estimate appropriate limits during ramp-up throttling.
    """
    mutate_fn = DeleteFromDatastore._DatastoreDeleteFn(project)
    super().__init__(mutate_fn, throttle_rampup, hint_num_workers)

  class _DatastoreDeleteFn(_Mutate.DatastoreMutateFn):
    def element_to_client_batch_item(self, element):
      if not isinstance(element, types.Key):
        raise ValueError(
            'apache_beam.io.gcp.datastore.v1new.datastoreio.Key'
            ' expected, got: %s' % type(element))
      if not element.project:
        element.project = self._project
      client_key = element.to_client_key()
      if client_key.is_partial:
        raise ValueError(
            'Keys to be deleted from Cloud Datastore must be '
            'complete:\n%s' % client_key)
      return client_key

    def add_to_batch(self, client_key):
      self._batch.delete(client_key)

    def display_data(self):
      return {
          'mutation': 'Delete',
          'project': self._project,

View on GitHub (pinned to 12126d8942)

Solutions

  1. Map each element to apache_beam.io.gcp.datastore.v1new.types.Key before DeleteFromDatastore.
  2. If you have Entities, use element.key to extract the v1new Key.
  3. If you have (kind, id) pairs, construct types.Key(kind, id, project=project) in a Map step.
  4. Verify imports come from datastore.v1new.types, not the legacy v1 package.

Example fix

// before
| 'delete' >> DeleteFromDatastore(project)   # elements are dicts
// after
| 'to_key' >> beam.Map(lambda d: types.Key(d['kind'], d['id'], project=project)) \
| 'delete' >> DeleteFromDatastore(project)
Defensive patterns

Strategy: type-guard

Validate before calling

from apache_beam.io.gcp.datastore.v1new import types
assert all(isinstance(k, types.Key) for k in elements), 'non-Key element for DeleteFromDatastore'

Type guard

def is_deletable_key(e):
    from apache_beam.io.gcp.datastore.v1new import types
    return isinstance(e, types.Key)

Try / catch

try:
    _ = pcoll | DeleteFromDatastore(project)
except ValueError as e:
    if 'Key expected' in str(e):
        pcoll = pcoll | beam.Map(lambda e: e.key if isinstance(e, types.Entity) else to_key(e))
        _ = pcoll | DeleteFromDatastore(project)

Prevention

When it happens

Trigger: Piping Entities, dicts, strings, or google.datastore.v1 Keys into DeleteFromDatastore; feeding output of a source that yields entity payloads instead of just keys; using v1-era key types with the v1new sink.

Common situations: Wanting to delete whole entities from a read pipeline (must project out keys first); mixing datastore v1 and v1new APIs; passing key tuples like (kind, id) directly.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ffe7b03eab1437ae. Report an issue: GitHub.