{"record":{"id":"1d1ed6bb6b2458e1","repo":"apache/beam","slug":"mltransform-may-have-been-utilized-alongside-transforms","errorCode":null,"errorMessage":"MLTransform may have been utilized alongside transforms written in TensorFlow Transform, in conjunction with those from different frameworks. Currently, retrieving artifacts from this multi-framework setup is not supported.","messagePattern":"MLTransform may have been utilized alongside transforms written in TensorFlow Transform, in conjunction with those from different frameworks\\. Currently, retrieving artifacts from this multi-framework setup is not supported\\.","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/transforms/utils.py","lineNumber":66,"sourceCode":"\n  This is intended to be used for testing purposes only.\n  \"\"\"\n  def __init__(self, artifact_location: str):\n    tempdir = tempfile.mkdtemp()\n    if artifact_location.startswith('gs://'):\n      parts = artifact_location[5:].split('/')\n      bucket_name = parts[0]\n      prefix = '/'.join(parts[1:])\n      download_artifacts_from_gcs(bucket_name, prefix, tempdir)\n\n    assert os.listdir(tempdir), f\"No files found in {artifact_location}\"\n    artifact_location = os.path.join(tempdir, prefix)\n    files = os.listdir(artifact_location)\n    files.remove(base._ATTRIBUTE_FILE_NAME)\n    # TODO: https://github.com/apache/beam/issues/29356\n    #  Integrate ArtifactFetcher into MLTransform.\n    if len(files) > 1:\n      raise NotImplementedError(\n          \"MLTransform may have been utilized alongside transforms written \"\n          \"in TensorFlow Transform, in conjunction with those from different \"\n          \"frameworks. Currently, retrieving artifacts from this \"\n          \"multi-framework setup is not supported.\")\n    self._artifact_location = os.path.join(artifact_location, files[0])\n    self.transform_output = tft.TFTransformOutput(self._artifact_location)\n\n  def get_vocab_list(self, vocab_filename: str) -> list[bytes]:\n    \"\"\"\n    Returns list of vocabulary terms created during MLTransform.\n    \"\"\"\n    try:\n      vocab_list = self.transform_output.vocabulary_by_name(vocab_filename)\n    except ValueError as e:\n      raise ValueError(\n          'Vocabulary file {} not found in artifact location'.format(\n              vocab_filename)) from e\n    return [x.decode('utf-8') for x in vocab_list]","sourceCodeStart":48,"sourceCodeEnd":84,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/transforms/utils.py#L48-L84","documentation":"ArtifactFetchingService fetches MLTransform artifacts from the artifact location, but if more than one framework artifact directory is found (besides the attribute file), it raises NotImplementedError. Multi-framework artifact retrieval (e.g. TF Transform plus another framework) is not supported yet (tracked by beam issue 29356).","triggerScenarios":"An MLTransform pipeline mixing TFT-based transforms with transforms from other frameworks (e.g. sklearn/torch wrappers), so the artifact tempdir contains multiple subdirectories; then ArtifactFetchingService.__init__ lists files and finds len(files) > 1.","commonSituations":"Combining tft.Scale/ComputeAndApplyVocabulary with non-TFT transforms like standardize/sentencepiece in the same MLTransform, then running with artifact fetching enabled (e.g. multi-worker scenarios).","solutions":["Use transforms from a single framework (all TFT-based) in one MLTransform.","Split the pipeline into multiple MLTransform stages, each with one framework's transforms.","Wait for/track apache/beam#29356 for multi-framework artifact support."],"exampleFix":"# before\nMLTransform(...).with_transform(tft.Scale(...)).with_transform(non_tft_transform)\n# after\nMLTransform(...).with_transform(tft.Scale(...))  # single framework only","handlingStrategy":"validation","validationCode":"# ensure all transforms are TFT-based before MLTransform with artifact fetching\nassert all(isinstance(t, TftTransform) for t in transforms), 'mixing frameworks unsupported'","typeGuard":null,"tryCatchPattern":"try:\n    fetcher = ArtifactFetchingService(artifact_location)\nexcept NotImplementedError as e:\n    logging.error('Use a single framework per MLTransform: %s', e)\n    raise","preventionTips":["Keep TFT and non-TFT transforms in separate MLTransform stages.","Check the beam issue tracker before mixing framework transforms."],"tags":["apache-beam","python","ml-transform","unsupported-operation"],"backgroundTag":"unsupported-operation","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T21:17:11.552Z"}