apache/beam · error · FileExistsError

The artifact location

Error message

The artifact location %s already exists and contains %s. Please specify a different location.

What it means

To avoid corrupting artifacts shared across Beam jobs, the JsonPickle attribute manager refuses to overwrite an existing attributes file. Before saving it checks FileSystems.exists(artifact_location/<attribute file>) and raises FileExistsError if the artifact file is already present.

Solutions

  1. Choose a new, unique artifact_location (e.g. include a run id/uuid) for this run.
  2. Manually delete the existing artifact directory (or just the attribute file) if the old artifacts are no longer needed.
  3. In CI/production, generate the artifact path per-run: artifact_location=os.path.join(base, uuid.uuid4().hex).

Example fix

// before
MLTransform(artifact_location='gs://bucket/artifacts')
// after
MLTransform(artifact_location=f'gs://bucket/artifacts/{uuid.uuid4().hex[:6]}')
Defensive patterns

Strategy: validation

Validate before calling

from apache_beam.io.filesystems import FileSystems
import os
if FileSystems.exists(FileSystems.join(artifact_location, 'attributes.json')):
    artifact_location = os.path.join(artifact_location, uuid.uuid4().hex[:6])

Try / catch

try:
    save_attributes(artifact_location)
except FileExistsError:
    artifact_location = os.path.join(artifact_location, uuid.uuid4().hex[:6])
    save_attributes(artifact_location)

Prevention

When it happens

Trigger: Re-running MLTransform with the same artifact_location that a previous run already populated; pointing two pipelines at the same artifact directory.

Common situations: Iterating on a pipeline without bumping/changing artifact_location; a failed run left a partial artifact directory behind; scheduled jobs reusing a fixed artifact path.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/10594580b615205b. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/transforms/base.py:554

    if not is_gcs and path.find('://') != -1:
      raise RuntimeError(
          "Artifact locations are currently supported for only available for "
          "local paths and GCS paths. Got: %s" % path)
    return is_gcs

  @staticmethod
  def save_attributes(
      ptransform_list,
      artifact_location,
      **kwargs,
  ):
    # if an artifact location is present, instead of overwriting the
    # existing file, raise an error since the same artifact location
    # can be used by multiple beam jobs and this could result in undesired
    # behavior.
    if FileSystems.exists(FileSystems.join(artifact_location,
                                           _ATTRIBUTE_FILE_NAME)):
      raise FileExistsError(
          "The artifact location %s already exists and contains %s. Please "
          "specify a different location." %
          (artifact_location, _ATTRIBUTE_FILE_NAME))

    if _JsonPickleTransformAttributeManager._is_remote_path(artifact_location):
      temp_dir = tempfile.mkdtemp()
      temp_json_file = os.path.join(temp_dir, _ATTRIBUTE_FILE_NAME)
      with open(temp_json_file, 'w+') as f:
        f.write(jsonpickle.encode(ptransform_list))
      with open(temp_json_file, 'rb') as f:
        from apache_beam.runners.dataflow.internal import apiclient
        _LOGGER.info('Creating artifact location: %s', artifact_location)
        # pipeline options required to for the client to configure project.
        options = kwargs.get('options')
        try:
          apiclient.DataflowApplicationClient(options=options).stage_file(
              gcs_or_local_path=artifact_location,
              file_name=_ATTRIBUTE_FILE_NAME,

View on GitHub (pinned to 12126d8942)