{"record":{"id":"2bef2658f5b4acff","repo":"apache/beam","slug":"redefinition-of-field-name-cannot-append-a-field-that","errorCode":null,"errorMessage":"Redefinition of field \"{name}\". Cannot append a field that already exists in original input.","messagePattern":"Redefinition of field \"(.+?)\"\\. Cannot append a field that already exists in original input\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":682,"sourceCode":"    input_schema = dict(named_fields_from_element_type(pcoll.element_type))\n  except (TypeError, ValueError) as exn:\n    if drop:\n      raise ValueError(\"Can only drop fields on a schema'd input.\") from exn\n    if append:\n      raise ValueError(\"Can only append fields on a schema'd input.\") from exn\n    elif any(is_expr(x) for x in fields.values()):\n      raise ValueError(\"Can only use expressions on a schema'd input.\") from exn\n    input_schema = {}\n\n  if drop and not append:\n    raise ValueError(\"Can only drop fields if append is true.\")\n  for name in drop:\n    if name not in input_schema:\n      raise ValueError(f'Dropping unknown field \"{name}\"')\n  if append:\n    for name in fields:\n      if name in input_schema and name not in drop:\n        raise ValueError(\n            f'Redefinition of field \"{name}\". '\n            'Cannot append a field that already exists in original input.')\n\n  if append:\n    return input_schema, {\n        **{name: f'`{name}`' if language in ['sql', 'calcite'] else name\n           for name in input_schema.keys() if name not in drop},\n        **fields\n    }\n  else:\n    return input_schema, fields\n\n\n@beam.ptransform.ptransform_fn\n@maybe_with_exception_handling_transform_fn\ndef _PyJsMapToFields(\n    pcoll,\n    fields: Mapping[str, Union[str, Mapping[str, str]]],","sourceCodeStart":664,"sourceCodeEnd":700,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L664-L700","documentation":"Raised by `normalize_fields` when `append: true` and the `fields` mapping tries to add a field whose name already exists in the input schema and is not being dropped. Appending is strictly additive; to overwrite an existing column you must also list it in `drop` (append-and-redefine) rather than silently replacing it.","triggerScenarios":"YAML AddFields/MapToFields config with `append: true` and a `fields` key that matches an existing input column, where that column is absent from the `drop` list. The check `if name in input_schema and name not in drop` fires.","commonSituations":"Attempting to overwrite/replace an existing column via AddFields; unaware that redefining requires drop; copy-pasting configs where the added field already exists upstream.","solutions":["Add the existing field name to the `drop` list so it is redefined rather than duplicated.","Rename the new field to a non-conflicting name.","Remove the conflicting field from `fields` if the existing column is already correct."],"exampleFix":"// before\nconfig:\n  append: true\n  fields:\n    score: {type: double, value: \"score * 2\"}\n// after\nconfig:\n  append: true\n  drop: [score]\n  fields:\n    score: {type: double, value: \"score * 2\"}","handlingStrategy":"validation","validationCode":"schema_fields = dict(named_fields_from_element_type(pcoll.element_type)).keys()\nconflicts = [n for n in fields if n in schema_fields and n not in drop]\nif conflicts:\n    raise ValueError(f'Appending existing fields without dropping them: {conflicts}; add them to drop.')","typeGuard":null,"tryCatchPattern":"try:\n    out = normalize_fields(pcoll, fields={'score': expr}, drop=[], append=True)\nexcept ValueError as e:\n    if 'Redefinition of field' in str(e):\n        out = normalize_fields(pcoll, fields={'score': expr}, drop=['score'], append=True)\n    else:\n        raise","preventionTips":["Check the input schema for name collisions before defining appended fields","Treat overwrite = drop + append in this transform's semantics","Prefer unique new field names to avoid silent shadowing"],"tags":["apache-beam","yaml","schema","configuration"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}