{"record":{"id":"890526e09129ab96","repo":"apache/beam","slug":"can-only-drop-fields-if-append-is-true","errorCode":null,"errorMessage":"Can only drop fields if append is true.","messagePattern":"Can only drop fields if append is true\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/yaml/yaml_mapping.py","lineNumber":675,"sourceCode":"\ndef is_expr(v):\n  return isinstance(v, str) or (isinstance(v, dict) and 'expression' in v)\n\n\ndef normalize_fields(pcoll, fields, drop=(), append=False, language='generic'):\n  try:\n    input_schema = dict(named_fields_from_element_type(pcoll.element_type))\n  except (TypeError, ValueError) as exn:\n    if drop:\n      raise ValueError(\"Can only drop fields on a schema'd input.\") from exn\n    if append:\n      raise ValueError(\"Can only append fields on a schema'd input.\") from exn\n    elif any(is_expr(x) for x in fields.values()):\n      raise ValueError(\"Can only use expressions on a schema'd input.\") from exn\n    input_schema = {}\n\n  if drop and not append:\n    raise ValueError(\"Can only drop fields if append is true.\")\n  for name in drop:\n    if name not in input_schema:\n      raise ValueError(f'Dropping unknown field \"{name}\"')\n  if append:\n    for name in fields:\n      if name in input_schema and name not in drop:\n        raise ValueError(\n            f'Redefinition of field \"{name}\". '\n            'Cannot append a field that already exists in original input.')\n\n  if append:\n    return input_schema, {\n        **{name: f'`{name}`' if language in ['sql', 'calcite'] else name\n           for name in input_schema.keys() if name not in drop},\n        **fields\n    }\n  else:\n    return input_schema, fields","sourceCodeStart":657,"sourceCodeEnd":693,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/yaml/yaml_mapping.py#L657-L693","documentation":"Raised by `normalize_fields` when `drop` is provided without `append: true`. In Beam YAML's field normalization, dropping fields is only meaningful as part of an append (redefine-and-drop) operation; dropping alone is rejected to force the caller to state intent explicitly.","triggerScenarios":"Invoking a YAML mapping transform with `drop: [field]` but no `append: true` in the config, after the schema lookup succeeded (input is schema'd). The check `if drop and not append` fires.","commonSituations":"Confusing the DropFields-like semantics: users expect a standalone drop, but this transform requires append alongside drop; copy-pasted configs missing the append flag.","solutions":["Add `append: true` to the transform config when using `drop`.","Use the dedicated DropFields YAML transform instead if you only want to remove fields.","Remove the `drop` option if dropping was not intended."],"exampleFix":"// before\n- type: AddFields\n  input: rows\n  config:\n    drop: [temp_col]\n// after\n- type: AddFields\n  input: rows\n  config:\n    append: true\n    drop: [temp_col]\n    fields: {}","handlingStrategy":"validation","validationCode":"if drop and not append:\n    raise ValueError('drop requires append: true in this transform; or use DropFields instead.')","typeGuard":null,"tryCatchPattern":"try:\n    out = normalize_fields(pcoll, fields={}, drop=['temp'])\nexcept ValueError as e:\n    if 'drop fields if append is true' in str(e):\n        out = normalize_fields(pcoll, fields={}, drop=['temp'], append=True)\n    else:\n        raise","preventionTips":["Remember drop is always paired with append in AddFields-style configs","Use the dedicated DropFields transform for standalone field removal","Lint YAML configs to enforce the drop/append pairing"],"tags":["apache-beam","yaml","configuration"],"backgroundTag":"conflicting-config-options","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}