apache/beam · error · ValueError

custom_gcs_temp_location must be str or ValueProvider

Error message

custom_gcs_temp_location must be str or ValueProvider

What it means

BigQueryBatchFileLoads stores the custom GCS staging location internally as a ValueProvider so it can be resolved at runtime. The __init__ accepts either a plain string or a ValueProvider; anything else (int, None with a non-empty-able mismatch, dict, etc.) falls through to this ValueError.

Solutions

1. Convert the value: custom_gcs_temp_location=str(my_path).
2. Wrap runtime values: custom_gcs_temp_location=RuntimeValueProvider(str, ...) or StaticValueProvider(str, value).
3. Omit the parameter to fall back to --temp_location.

Example fix

// before
beam.io.WriteToBigQuery(table, custom_gcs_temp_location=pathlib.Path('gs://bucket/temp'))
// after
beam.io.WriteToBigQuery(table, custom_gcs_temp_location=str(pathlib.Path('gs://bucket/temp')))

if custom_gcs_temp_location is not None and not isinstance(custom_gcs_temp_location, (str, vp.ValueProvider)):
    raise TypeError('custom_gcs_temp_location must be str or ValueProvider')

def is_valid_temp_location(v) -> bool:
    return v is None or isinstance(v, (str, vp.ValueProvider))

When it happens

Trigger: Calling WriteToBigQuery/BigQueryBatchFileLoads with custom_gcs_temp_location set to a non-str, non-ValueProvider object (e.g. bytes, int, Path object).

Common situations: Passing a pathlib.Path or os.PathLike bucket path instead of str, or wrapping the location incorrectly when plumbing runtime parameters.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• temp_dataset has to be either str or DatasetReference
• Does not support converting unknown type value: " +…
• Error converting field :
• is not primitive type.
• Problem converting field

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ce1c30d0cdc02a5c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_file_loads.py:997

    self.project = project
    self.create_disposition = create_disposition
    self.write_disposition = write_disposition
    self.triggering_frequency = triggering_frequency
    self.with_auto_sharding = with_auto_sharding
    self.max_file_size = max_file_size or _DEFAULT_MAX_FILE_SIZE
    self.max_files_per_bundle = (
        max_files_per_bundle or _DEFAULT_MAX_WRITERS_PER_BUNDLE)
    self.max_partition_size = max_partition_size or _MAXIMUM_LOAD_SIZE
    self.max_files_per_partition = (
        max_files_per_partition or _MAXIMUM_SOURCE_URIS)
    if (isinstance(custom_gcs_temp_location, str) or
        custom_gcs_temp_location is None):
      self._custom_gcs_temp_location = vp.StaticValueProvider(
          str, custom_gcs_temp_location or '')
    elif isinstance(custom_gcs_temp_location, vp.ValueProvider):
      self._custom_gcs_temp_location = custom_gcs_temp_location
    else:
      raise ValueError('custom_gcs_temp_location must be str or ValueProvider')

    self.test_client = test_client
    self.schema = schema
    self._temp_file_format = temp_file_format or bigquery_tools.FileFormat.JSON

    # If we have multiple destinations, then we will have multiple load jobs,
    # thus we will need temporary tables for atomicity.
    self.dynamic_destinations = bool(callable(destination))

    self.additional_bq_parameters = additional_bq_parameters or {}
    self.table_side_inputs = table_side_inputs or ()
    self.schema_side_inputs = schema_side_inputs or ()

    self.is_streaming_pipeline = is_streaming_pipeline
    self.load_job_project_id = load_job_project_id
    self._validate = validate
    if self._validate:
      self.verify()

View on GitHub (pinned to 12126d8942)