apache/beam · error · NotImplementedError

MLTransform may have been utilized alongside transforms writ

Error message

MLTransform may have been utilized alongside transforms written in TensorFlow Transform, in conjunction with those from different frameworks. Currently, retrieving artifacts from this multi-framework setup is not supported.

What it means

ArtifactFetchingService fetches MLTransform artifacts from the artifact location, but if more than one framework artifact directory is found (besides the attribute file), it raises NotImplementedError. Multi-framework artifact retrieval (e.g. TF Transform plus another framework) is not supported yet (tracked by beam issue 29356).

Source

Thrown at sdks/python/apache_beam/ml/transforms/utils.py:66

  This is intended to be used for testing purposes only.
  """
  def __init__(self, artifact_location: str):
    tempdir = tempfile.mkdtemp()
    if artifact_location.startswith('gs://'):
      parts = artifact_location[5:].split('/')
      bucket_name = parts[0]
      prefix = '/'.join(parts[1:])
      download_artifacts_from_gcs(bucket_name, prefix, tempdir)

    assert os.listdir(tempdir), f"No files found in {artifact_location}"
    artifact_location = os.path.join(tempdir, prefix)
    files = os.listdir(artifact_location)
    files.remove(base._ATTRIBUTE_FILE_NAME)
    # TODO: https://github.com/apache/beam/issues/29356
    #  Integrate ArtifactFetcher into MLTransform.
    if len(files) > 1:
      raise NotImplementedError(
          "MLTransform may have been utilized alongside transforms written "
          "in TensorFlow Transform, in conjunction with those from different "
          "frameworks. Currently, retrieving artifacts from this "
          "multi-framework setup is not supported.")
    self._artifact_location = os.path.join(artifact_location, files[0])
    self.transform_output = tft.TFTransformOutput(self._artifact_location)

  def get_vocab_list(self, vocab_filename: str) -> list[bytes]:
    """
    Returns list of vocabulary terms created during MLTransform.
    """
    try:
      vocab_list = self.transform_output.vocabulary_by_name(vocab_filename)
    except ValueError as e:
      raise ValueError(
          'Vocabulary file {} not found in artifact location'.format(
              vocab_filename)) from e
    return [x.decode('utf-8') for x in vocab_list]

View on GitHub (pinned to 12126d8942)

Solutions

Use transforms from a single framework (all TFT-based) in one MLTransform.
Split the pipeline into multiple MLTransform stages, each with one framework's transforms.
Wait for/track apache/beam#29356 for multi-framework artifact support.

Example fix

# before
MLTransform(...).with_transform(tft.Scale(...)).with_transform(non_tft_transform)
# after
MLTransform(...).with_transform(tft.Scale(...))  # single framework only

Defensive patterns

Strategy: validation

Validate before calling

# ensure all transforms are TFT-based before MLTransform with artifact fetching
assert all(isinstance(t, TftTransform) for t in transforms), 'mixing frameworks unsupported'

Try / catch

try:
    fetcher = ArtifactFetchingService(artifact_location)
except NotImplementedError as e:
    logging.error('Use a single framework per MLTransform: %s', e)
    raise

Prevention

Keep TFT and non-TFT transforms in separate MLTransform stages.
Check the beam issue tracker before mixing framework transforms.

When it happens

Trigger: An MLTransform pipeline mixing TFT-based transforms with transforms from other frameworks (e.g. sklearn/torch wrappers), so the artifact tempdir contains multiple subdirectories; then ArtifactFetchingService.__init__ lists files and finds len(files) > 1.

Common situations: Combining tft.Scale/ComputeAndApplyVocabulary with non-TFT transforms like standardize/sentencepiece in the same MLTransform, then running with artifact fetching enabled (e.g. multi-worker scenarios).

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1d1ed6bb6b2458e1. Report an issue: GitHub.