apache/beam · error · ValueError

Either columns or type_adapter must be specified

Error message

Either columns or type_adapter must be specified

What it means

MLTransform's setup requires either a columns specification (used to build a dict adapter) or an explicit type_adapter; with neither, it cannot convert pipeline elements into the dict form transforms need, so a ValueError is raised.

Solutions

Pass columns=[...] to the transform configuration listing the dict keys to extract.
Alternatively supply a custom type_adapter implementing the expected conversion.
Check the MLTransform docs example for the with_transform(columns=...) pattern.

Example fix

// before
MLTransform(write_artifact_location=loc).with_transform(ImageEmbedding(model_name='resnet'))
// after
MLTransform(write_artifact_location=loc).with_transform(ImageEmbedding(model_name='resnet', columns=['image_bytes']))

Defensive patterns

Strategy: validation

Validate before calling

def build_mltransform(transforms, columns=None, type_adapter=None, **kw):
    if columns is None and type_adapter is None:
        raise ValueError('Pass columns or type_adapter')
    return MLTransform(**kw).with_transform(transforms(columns=columns)) if columns else MLTransform(**kw).with_transform(transforms(type_adapter=type_adapter))

Type guard

def transform_config_ok(cfg) -> bool:
    return bool(cfg.get('columns')) or cfg.get('type_adapter') is not None

Try / catch

try:
    t = MLTransform(...).with_transform(Embedding(model_handler=h, columns=cols))
except ValueError as e:
    if 'Either columns or type_adapter' in str(e):
        raise ValueError('Embedding transform requires columns= or type_adapter=') from e
    raise

Prevention

Always include columns= in every with_transform call
Lint pipeline code for MLTransform configs missing columns
Follow the official MLTransform example when scaffolding new pipelines

When it happens

Trigger: Constructing MLTransform (or a transform's with_transform configuration) without passing columns and without providing type_adapter.

Common situations: Copy-pasting MLTransform setup where the columns kwarg was accidentally deleted; custom transforms passing a model handler but forgetting input configuration.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/06d3f3ff6fa3cfeb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/transforms/base.py:281

      columns: Optional[list[str]] = None,
      type_adapter: Optional[EmbeddingTypeAdapter] = None,
      # common args for all ModelHandlers.
      load_model_args: Optional[dict[str, Any]] = None,
      min_batch_size: Optional[int] = None,
      max_batch_size: Optional[int] = None,
      large_model: bool = False,
      **kwargs):
    self.load_model_args = load_model_args or {}
    self.min_batch_size = min_batch_size
    self.max_batch_size = max_batch_size
    self.large_model = large_model
    self.columns = columns
    if columns is not None:
      self.type_adapter = _create_dict_adapter(columns)
    elif type_adapter is not None:
      self.type_adapter = type_adapter
    else:
      raise ValueError("Either columns or type_adapter must be specified")
    self.inference_args = kwargs.pop('inference_args', {})

    if kwargs:
      _LOGGER.warning("Ignoring the following arguments: %s", kwargs.keys())

  # TODO:https://github.com/apache/beam/pull/29564 add set_model_handler method
  @abc.abstractmethod
  def get_model_handler(self) -> ModelHandler:
    """
    Return framework specific model handler.
    """

  def get_columns_to_apply(self):
    return self.columns


class MLTransform(
    beam.PTransform[beam.PCollection[ExampleT],

View on GitHub (pinned to 12126d8942)