apache/beam · error · ValueError

apache_beam.io.gcp.datastore.v1new.datastoreio.Entity expect

Error message

apache_beam.io.gcp.datastore.v1new.datastoreio.Entity expected, got: %s

What it means

WriteToDatastore's _DatastoreWriteFn only accepts apache_beam.io.gcp.datastore.v1new.types.Entity elements; any other element type raises ValueError naming the actual type received. This is an eager type check before converting the element to a client entity.

Source

Thrown at sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py:538

      self,
      project,
      throttle_rampup=True,
      hint_num_workers=_Mutate._DEFAULT_HINT_NUM_WORKERS):
    """Initialize the `WriteToDatastore` transform.

    Args:
      project: (:class:`str`) The ID of the project to write entities to.
      throttle_rampup: Whether to enforce a gradual ramp-up.
      hint_num_workers: A hint for the expected number of workers, used to
                        estimate appropriate limits during ramp-up throttling.
    """
    mutate_fn = WriteToDatastore._DatastoreWriteFn(project)
    super().__init__(mutate_fn, throttle_rampup, hint_num_workers)

  class _DatastoreWriteFn(_Mutate.DatastoreMutateFn):
    def element_to_client_batch_item(self, element):
      if not isinstance(element, types.Entity):
        raise ValueError(
            'apache_beam.io.gcp.datastore.v1new.datastoreio.Entity'
            ' expected, got: %s' % type(element))
      if not element.key.project:
        element.key.project = self._project
      client_entity = element.to_client_entity()
      if client_entity.key.is_partial:
        raise ValueError(
            'Entities to be written to Cloud Datastore must '
            'have complete keys:\n%s' % client_entity)
      return client_entity

    def add_to_batch(self, client_entity):
      self._batch.put(client_entity)

    def display_data(self):
      return {
          'mutation': 'Write (upsert)',
          'project': self._project,

View on GitHub (pinned to 12126d8942)

Solutions

Wrap/convert each element to apache_beam.io.gcp.datastore.v1new.types.Entity before writing (Entity(key=Key(...), properties=...)).
If elements are google.cloud.datastore entities, convert their key and properties into v1new types.Entity.
Check the upstream PCollection's element type; add an explicit Map step that constructs the correct Entity.
Make sure you did not import types from datastore.v1 instead of datastore.v1new.

Example fix

// before
| 'write' >> WriteToDatastore(project)   # elements are dicts
// after
| 'to_entity' >> beam.Map(lambda d: types.Entity(
      key=types.Key(d['kind'], d['id'], project=project), properties=d['props'])) \
| 'write' >> WriteToDatastore(project)

Defensive patterns

Strategy: type-guard

Validate before calling

from apache_beam.io.gcp.datastore.v1new import types
assert all(isinstance(e, types.Entity) for e in elements), 'non-Entity element in PCollection'

Type guard

def is_writable_entity(e):
    from apache_beam.io.gcp.datastore.v1new import types
    return isinstance(e, types.Entity) and e.key is not None

Try / catch

try:
    _ = write_result = (pcoll | WriteToDatastore(project))
except ValueError as e:
    if 'Entity expected' in str(e):
        pcoll = pcoll | beam.Map(to_v1new_entity)
        write_result = pcoll | WriteToDatastore(project)

Prevention

Import Entity/Key from datastore.v1new.types, not the legacy v1 package
Add a beam.Map conversion step between your source and WriteToDatastore
Check PCollection element types with type hints (beam.PCollection[types.Entity])
Never feed raw dicts or google.cloud.datastore entities to v1new sinks

When it happens

Trigger: Piping non-Entity elements (dicts, google.cloud.datastore Entity objects, strings) into WriteToDatastore; using the older v1 types.Entity with the v1new sink (or vice versa); PCollection produced by a source that yields dicts.

Common situations: Migrating from apache_beam.io.gcp.datastore.v1 to v1new without converting element types; building Entities manually as dicts; mixing outputs of ReadFromDatastore (which yields v1new Entities) with hand-constructed elements of the wrong library.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a8937f46ce70873a. Report an issue: GitHub.