apache/beam · error · ValueError

Entities to be written to Cloud Datastore must have complete

Error message

Entities to be written to Cloud Datastore must have complete keys:
%s

What it means

WriteToDatastore requires every Entity to have a complete (fully specified) key — Datastore cannot upsert an entity whose key ends in an auto-id placeholder. The check happens after converting to a client entity and inspecting client_entity.key.is_partial.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:545

      project: (:class:`str`) The ID of the project to write entities to.
      throttle_rampup: Whether to enforce a gradual ramp-up.
      hint_num_workers: A hint for the expected number of workers, used to
                        estimate appropriate limits during ramp-up throttling.
    """
    mutate_fn = WriteToDatastore._DatastoreWriteFn(project)
    super().__init__(mutate_fn, throttle_rampup, hint_num_workers)

  class _DatastoreWriteFn(_Mutate.DatastoreMutateFn):
    def element_to_client_batch_item(self, element):
      if not isinstance(element, types.Entity):
        raise ValueError(
            'apache_beam.io.gcp.datastore.v1new.datastoreio.Entity'
            ' expected, got: %s' % type(element))
      if not element.key.project:
        element.key.project = self._project
      client_entity = element.to_client_entity()
      if client_entity.key.is_partial:
        raise ValueError(
            'Entities to be written to Cloud Datastore must '
            'have complete keys:\n%s' % client_entity)
      return client_entity

    def add_to_batch(self, client_entity):
      self._batch.put(client_entity)

    def display_data(self):
      return {
          'mutation': 'Write (upsert)',
          'project': self._project,
      }


@typehints.with_input_types(types.Key)
class DeleteFromDatastore(_Mutate):
  """
  Deletes elements matching input

View on GitHub (pinned to 12126d8942)

Solutions

Assign a complete key: provide an id or name (types.Key(kind, id_or_name, project=...)).
Pre-generate unique IDs in the pipeline (e.g. uuid or a hash of the entity contents) before writing.
If auto-ID allocation is required, do it in a separate step (e.g. allocate_ids or a non-Beam write path) — the Beam sink intentionally rejects partial keys.

Example fix

// before
ent = types.Entity(key=types.Key('Person', project=project))  # partial key
// after
ent = types.Entity(key=types.Key('Person', 12345, project=project))  # complete key

Defensive patterns

Strategy: validation

Validate before calling

def assert_complete_key(entity, project):
    from apache_beam.io.gcp.datastore.v1new import types
    if not isinstance(entity, types.Entity):
        raise TypeError('need types.Entity')
    if entity.key.id is None and entity.key.name is None:
        raise ValueError('partial key: assign id or name before writing')
    if not entity.key.project:
        entity.key.project = project

Type guard

def has_complete_key(entity):
    return bool(entity) and entity.key is not None and (entity.key.id is not None or entity.key.name is not None)

Try / catch

try:
    results |= WriteToDatastore(project)
except ValueError as e:
    if 'complete keys' in str(e):
        raise RuntimeError('pipeline emitted entities with partial keys; fix key construction') from e

Prevention

Always pass id or name when constructing types.Key
Remember the Beam sink does NOT auto-allocate IDs for partial keys
Generate deterministic IDs (hash/uuid) in the pipeline for new entities
Validate keys in a Map step before the sink in dev/test runs

When it happens

Trigger: Writing a v1new types.Entity whose Key has no id/name on its final path element (partial key), expecting Datastore to auto-allocate an ID during a Beam write.

Common situations: Constructing Key(kind) without id/name for inserts; assuming the sink behaves like google.cloud.datastore Client.put which supports partial keys for auto-ID allocation; copy-pasted key-construction code omitting the id parameter.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b891a7d40d038783. Report an issue: GitHub.