{"record":{"id":"6f1b961b5ab537ca","repo":"apache/beam","slug":"data-does-not-contain-the-following-columns-missing-columns","errorCode":null,"errorMessage":"Data does not contain the following columns : {missing_columns}.","messagePattern":"Data does not contain the following columns : (.+?)\\.","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/transforms/base.py","lineNumber":210,"sourceCode":"    raise TypeError(\n        'Expected data to be dicts, got '\n        f'{type(batch[0])} instead.')\n  result = []\n  expected_keys = set(batch[0].keys())\n  expected_columns = set(columns)\n  # Process one batch item at a time\n  for item in batch:\n    item_keys = item.keys() if isinstance(item, dict) else set()\n    if set(item_keys) != expected_keys:\n      extra_keys = item_keys - expected_keys\n      missing_keys = expected_keys - item_keys\n      raise RuntimeError(\n          f'All dicts in batch must have the same keys. '\n          f'extra keys: {extra_keys}, '\n          f'missing keys: {missing_keys}')\n    missing_columns = expected_columns - item_keys\n    if (missing_columns):\n      raise RuntimeError(\n          f'Data does not contain the following columns '\n          f': {missing_columns}.')\n\n    # Get all columns for this item\n    for col in columns:\n      if isinstance(item, dict):\n        result.append(item[col])\n  return result\n\n\ndef _dict_output_fn(\n    columns: Sequence[str],\n    batch: Sequence[Union[dict[str, Any], beam.Row]],\n    embeddings: Sequence[Any]) -> list[Union[dict[str, Any], beam.Row]]:\n  \"\"\"Map embeddings back to columns in batch.\"\"\"\n  is_beam_row = False\n  if batch and hasattr(batch[0], '_asdict'):\n    is_beam_row = True","sourceCodeStart":192,"sourceCodeEnd":228,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/transforms/base.py#L192-L228","documentation":"Raised from the batch-processing helper behind MLTransform's dict columns: one or more column names listed for the transform are absent from the input dicts, so the requested per-column operation has nothing to apply to. The missing column names are interpolated into the message.","triggerScenarios":"Specifying columns=['embedding_input'] while the input dicts only have a 'text' key; typos in column names; transforms configured with columns that the upstream data never produces.","commonSituations":"Renaming fields upstream without updating MLTransform(columns=...); copying example code whose column names differ from your data schema.","solutions":["Align the columns argument with the actual keys in your dicts (or rename dict keys to match columns).","Add a beam.Map to populate missing columns with defaults before the transform.","Print/log a sample element's keys to verify names before configuring columns."],"exampleFix":"// before\nMLTransform(...).with_transform(t(columns=['sentence']))\n// after\nMLTransform(...).with_transform(t(columns=['text']))  # 'text' is the real key","handlingStrategy":"validation","validationCode":"required = {'text'}\nsample = next(iter(pcoll), None)\nassert sample is None or required <= set(sample.keys()), f'Missing columns: {required - set(sample.keys())}'","typeGuard":"def has_columns(d, columns) -> bool:\n    return isinstance(d, dict) and set(columns) <= set(d.keys())","tryCatchPattern":"try:\n    out = data | MLTransform(...)\nexcept RuntimeError as e:\n    if 'does not contain the following columns' in str(e):\n        missing = str(e).split(': ')[-1]\n        data = data | beam.Map(lambda d, m=missing.strip(\"{}\"): {k.strip(\" '\"): None for k in m.split(',') if k.strip(\" '\") not in d} | d)\n        out = data | MLTransform(...)","preventionTips":["Log a sample element's keys before configuring columns","Keep column names in a shared constant used by producers and transforms","Rename upstream fields and columns in the same commit"],"tags":["python","apache-beam","schema"],"backgroundTag":"missing-required-config-field","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}