{"record":{"id":"ac5f36b00b513c15","repo":"apache/beam","slug":"failed-to-create-dataflow-client-pipeline-options-are","errorCode":null,"errorMessage":"Failed to create Dataflow client. Pipeline options are required to save the attributes.in the artifact location %s","messagePattern":"Failed to create Dataflow client\\. Pipeline options are required to save the attributes\\.in the artifact location (.+?)","errorType":"exception","errorClass":"RuntimeError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/ml/transforms/base.py","lineNumber":577,"sourceCode":"    if _JsonPickleTransformAttributeManager._is_remote_path(artifact_location):\n      temp_dir = tempfile.mkdtemp()\n      temp_json_file = os.path.join(temp_dir, _ATTRIBUTE_FILE_NAME)\n      with open(temp_json_file, 'w+') as f:\n        f.write(jsonpickle.encode(ptransform_list))\n      with open(temp_json_file, 'rb') as f:\n        from apache_beam.runners.dataflow.internal import apiclient\n        _LOGGER.info('Creating artifact location: %s', artifact_location)\n        # pipeline options required to for the client to configure project.\n        options = kwargs.get('options')\n        try:\n          apiclient.DataflowApplicationClient(options=options).stage_file(\n              gcs_or_local_path=artifact_location,\n              file_name=_ATTRIBUTE_FILE_NAME,\n              stream=f,\n              mime_type='application/json')\n        except Exception as exc:\n          if not options:\n            raise RuntimeError(\n                \"Failed to create Dataflow client. \"\n                \"Pipeline options are required to save the attributes.\"\n                \"in the artifact location %s\" % artifact_location) from exc\n          raise\n    else:\n      if not FileSystems.exists(artifact_location):\n        FileSystems.mkdirs(artifact_location)\n      # FileSystems.open() fails if the file does not exist.\n      with open(os.path.join(artifact_location, _ATTRIBUTE_FILE_NAME),\n                'w+') as f:\n        f.write(jsonpickle.encode(ptransform_list))\n\n  @staticmethod\n  def load_attributes(artifact_location):\n    with FileSystems.open(os.path.join(artifact_location, _ATTRIBUTE_FILE_NAME),\n                          'rb') as f:\n      return jsonpickle.decode(f.read())\n","sourceCodeStart":559,"sourceCodeEnd":595,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/ml/transforms/base.py#L559-L595","documentation":"When saving attributes to a GCS path, the manager may use a Dataflow-based GCS upload helper that needs pipeline options to construct its client. If an exception occurs in that helper and no pipeline options were supplied, the original error is re-raised as a RuntimeError explaining that pipeline options are required to save attributes to the artifact location.","triggerScenarios":"Saving MLTransform attributes to a gs:// artifact_location without passing PipelineOptions (e.g. options=None), so the GCS/Dataflow client cannot be created (no project/credentials context).","commonSituations":"Running MLTransform in a standalone script or unit test writing to GCS without providing --project/--temp_location options; missing default credentials so the Dataflow/GCS client construction fails.","solutions":["Pass PipelineOptions to MLTransform (e.g. PipelineOptions(['--project=my-project'])) when artifact_location is a GCS path.","Run with proper Google Cloud credentials (GOOGLE_APPLICATION_CREDENTIALS or ADC) so the client can be constructed.","Save artifacts to a local path instead if you do not need GCS.","Fix the underlying chained exception (inspect the 'from exc' cause) such as network or permission errors."],"exampleFix":"// before\nMLTransform(artifact_location='gs://bucket/artifacts')\n// after\nfrom apache_beam.options.pipeline_options import PipelineOptions\nMLTransform(artifact_location='gs://bucket/artifacts',\n            options=PipelineOptions(['--project=my-project']))","handlingStrategy":"validation","validationCode":"from apache_beam.options.pipeline_options import PipelineOptions\nif artifact_location.startswith('gs://') and options is None:\n    options = PipelineOptions(['--project=my-project'])","typeGuard":null,"tryCatchPattern":"try:\n    save_attributes(gcs_path)\nexcept RuntimeError as e:\n    if 'Pipeline options are required' in str(e):\n        save_attributes(gcs_path, options=PipelineOptions(['--project=my-project']))\n    else:\n        raise","preventionTips":["Always pass PipelineOptions when artifact_location is a GCS path.","Ensure GOOGLE_APPLICATION_CREDENTIALS / ADC is configured in all environments.","Provide --project (and temp_location) in worker environment options.","Inspect the chained exception (from exc) to catch credential issues early."],"tags":["python","gcs","pipeline-options","artifact-location"],"backgroundTag":"missing-required-config","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}