apache/beam · error · ValueError

Keys to be deleted from Cloud Datastore must be complete: %s

Error message

Keys to be deleted from Cloud Datastore must be complete:
%s

What it means

DeleteFromDatastore requires every Key to be complete — a partial key (missing the final id/name) does not uniquely identify an entity, so the sink refuses to delete it, raising ValueError after checking client_key.is_partial.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:598

        be deleted.
      throttle_rampup: Whether to enforce a gradual ramp-up.
      hint_num_workers: A hint for the expected number of workers, used to
                        estimate appropriate limits during ramp-up throttling.
    """
    mutate_fn = DeleteFromDatastore._DatastoreDeleteFn(project)
    super().__init__(mutate_fn, throttle_rampup, hint_num_workers)

  class _DatastoreDeleteFn(_Mutate.DatastoreMutateFn):
    def element_to_client_batch_item(self, element):
      if not isinstance(element, types.Key):
        raise ValueError(
            'apache_beam.io.gcp.datastore.v1new.datastoreio.Key'
            ' expected, got: %s' % type(element))
      if not element.project:
        element.project = self._project
      client_key = element.to_client_key()
      if client_key.is_partial:
        raise ValueError(
            'Keys to be deleted from Cloud Datastore must be '
            'complete:\n%s' % client_key)
      return client_key

    def add_to_batch(self, client_key):
      self._batch.delete(client_key)

    def display_data(self):
      return {
          'mutation': 'Delete',
          'project': self._project,
      }

View on GitHub (pinned to 12126d8942)

Solutions

Supply the full key including id or name: types.Key(kind, id_or_name, project=project).
Fix the upstream data extraction so the identifying field is present (check for None/missing ids before the sink).
Filter out or dead-letter elements with incomplete keys using a validation step prior to DeleteFromDatastore.

Example fix

// before
key = types.Key('Person', project=project)
| 'delete' >> DeleteFromDatastore(project)
// after
key = types.Key('Person', record['person_id'], project=project)
| 'delete' >> DeleteFromDatastore(project)

Defensive patterns

Strategy: validation

Validate before calling

def assert_complete_delete_key(key):
    from apache_beam.io.gcp.datastore.v1new import types
    if not isinstance(key, types.Key):
        raise TypeError('need types.Key')
    if key.id is None and key.name is None:
        raise ValueError('cannot delete: key is partial (no id/name)')

Type guard

def is_complete_key(key):
    return isinstance(key, types.Key) and (key.id is not None or key.name is not None)

Try / catch

bad = pcoll | 'filter_partial' >> beam.Filter(lambda k: k.id is None and k.name is None)
try:
    _ = pcoll | DeleteFromDatastore(project)
except ValueError as e:
    if 'must be complete' in str(e):
        raise RuntimeError('partial keys reached delete sink') from e

Prevention

Verify source records actually contain the id/name before building keys
Drop or dead-letter partial keys with beam.Filter before the sink
Log and monitor filtered-out partial keys to catch upstream schema drift
Never assume wild-card deletes on partial keys

When it happens

Trigger: Deleting a v1new types.Key built without an id/name on its last path element, e.g. types.Key('Person', project=project), or a key read from an entity whose key was never finalized.

Common situations: Constructing delete keys from incomplete records; missing the id field in input data due to schema changes; assuming wild-card/partial-key deletes are supported (they are not in the Beam sink).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/65403a3b0f174e43. Report an issue: GitHub.