{"record":{"id":"b891a7d40d038783","repo":"apache/beam","slug":"entities-to-be-written-to-cloud-datastore-must-have-complete","errorCode":null,"errorMessage":"Entities to be written to Cloud Datastore must have complete keys:\n%s","messagePattern":"Entities to be written to Cloud Datastore must have complete keys:\n(.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py","lineNumber":545,"sourceCode":"      project: (:class:`str`) The ID of the project to write entities to.\n      throttle_rampup: Whether to enforce a gradual ramp-up.\n      hint_num_workers: A hint for the expected number of workers, used to\n                        estimate appropriate limits during ramp-up throttling.\n    \"\"\"\n    mutate_fn = WriteToDatastore._DatastoreWriteFn(project)\n    super().__init__(mutate_fn, throttle_rampup, hint_num_workers)\n\n  class _DatastoreWriteFn(_Mutate.DatastoreMutateFn):\n    def element_to_client_batch_item(self, element):\n      if not isinstance(element, types.Entity):\n        raise ValueError(\n            'apache_beam.io.gcp.datastore.v1new.datastoreio.Entity'\n            ' expected, got: %s' % type(element))\n      if not element.key.project:\n        element.key.project = self._project\n      client_entity = element.to_client_entity()\n      if client_entity.key.is_partial:\n        raise ValueError(\n            'Entities to be written to Cloud Datastore must '\n            'have complete keys:\\n%s' % client_entity)\n      return client_entity\n\n    def add_to_batch(self, client_entity):\n      self._batch.put(client_entity)\n\n    def display_data(self):\n      return {\n          'mutation': 'Write (upsert)',\n          'project': self._project,\n      }\n\n\n@typehints.with_input_types(types.Key)\nclass DeleteFromDatastore(_Mutate):\n  \"\"\"\n  Deletes elements matching input","sourceCodeStart":527,"sourceCodeEnd":563,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/gcp/datastore/v1new/datastoreio.py#L527-L563","documentation":"WriteToDatastore requires every Entity to have a complete (fully specified) key — Datastore cannot upsert an entity whose key ends in an auto-id placeholder. The check happens after converting to a client entity and inspecting client_entity.key.is_partial.","triggerScenarios":"Writing a v1new types.Entity whose Key has no id/name on its final path element (partial key), expecting Datastore to auto-allocate an ID during a Beam write.","commonSituations":"Constructing Key(kind) without id/name for inserts; assuming the sink behaves like google.cloud.datastore Client.put which supports partial keys for auto-ID allocation; copy-pasted key-construction code omitting the id parameter.","solutions":["Assign a complete key: provide an id or name (types.Key(kind, id_or_name, project=...)).","Pre-generate unique IDs in the pipeline (e.g. uuid or a hash of the entity contents) before writing.","If auto-ID allocation is required, do it in a separate step (e.g. allocate_ids or a non-Beam write path) — the Beam sink intentionally rejects partial keys."],"exampleFix":"// before\nent = types.Entity(key=types.Key('Person', project=project))  # partial key\n// after\nent = types.Entity(key=types.Key('Person', 12345, project=project))  # complete key","handlingStrategy":"validation","validationCode":"def assert_complete_key(entity, project):\n    from apache_beam.io.gcp.datastore.v1new import types\n    if not isinstance(entity, types.Entity):\n        raise TypeError('need types.Entity')\n    if entity.key.id is None and entity.key.name is None:\n        raise ValueError('partial key: assign id or name before writing')\n    if not entity.key.project:\n        entity.key.project = project","typeGuard":"def has_complete_key(entity):\n    return bool(entity) and entity.key is not None and (entity.key.id is not None or entity.key.name is not None)","tryCatchPattern":"try:\n    results |= WriteToDatastore(project)\nexcept ValueError as e:\n    if 'complete keys' in str(e):\n        raise RuntimeError('pipeline emitted entities with partial keys; fix key construction') from e","preventionTips":["Always pass id or name when constructing types.Key","Remember the Beam sink does NOT auto-allocate IDs for partial keys","Generate deterministic IDs (hash/uuid) in the pipeline for new entities","Validate keys in a Map step before the sink in dev/test runs"],"tags":["python","datastore","validation","beam-io"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}