apache/beam · error · IllegalArgumentException

BigQuery temp location expected a valid 'gs://' path, but wa

Error message

BigQuery temp location expected a valid 'gs://' path, but was given '%s'

What it means

BatchLoads.validate() requires the BigQuery sink's temp location to be a valid 'gs://' GCS URI whenever custom (non-builtin) BigQuery services are used. It parses tempLocation with GcsPath.fromUri and rethrows any IllegalArgumentException with a message telling the user to set it via withCustomGcsTempLocation() or the --tempLocation pipeline option.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BatchLoads.java:310

      if (!customGcsTempLocation.isAccessible()) {
        // Can't perform verification in this case.
        return;
      } else if (Strings.isNullOrEmpty(customGcsTempLocation.get())) {
        tempLocation = options.getTempLocation();
      } else {
        tempLocation = customGcsTempLocation.get();
      }
    }
    checkArgument(
        !Strings.isNullOrEmpty(tempLocation),
        "BigQueryIO.Write needs a GCS temp location to store temp files. "
            + "This can be set by withCustomGcsTempLocation() in the Builder "
            + "or through the fallback pipeline option --tempLocation.");
    if (bigQueryServices == null) {
      try {
        GcsPath.fromUri(tempLocation);
      } catch (IllegalArgumentException e) {
        throw new IllegalArgumentException(
            String.format(
                "BigQuery temp location expected a valid 'gs://' path, but was given '%s'",
                tempLocation),
            e);
      }
    }
  }

  // Expand the pipeline when the user has requested periodically-triggered file writes.
  private WriteResult expandTriggered(PCollection<KV<DestinationT, ElementT>> input) {
    Duration triggeringFrequency = Preconditions.checkStateNotNull(this.triggeringFrequency);
    Pipeline p = input.getPipeline();
    final PCollectionView<String> loadJobIdPrefixView = createJobIdPrefixView(p, JobType.LOAD);
    final PCollectionView<String> tempLoadJobIdPrefixView =
        createJobIdPrefixView(p, JobType.TEMP_TABLE_LOAD);
    final PCollectionView<String> zeroLoadJobIdPrefixView =
        createJobIdPrefixView(p, JobType.SCHEMA_UPDATE);
    final PCollectionView<String> copyJobIdPrefixView = createJobIdPrefixView(p, JobType.COPY);

View on GitHub (pinned to 12126d8942)

Solutions

Set --tempLocation=gs://<bucket>/<path> in your pipeline options
Or call BigQueryIO.write().withCustomGcsTempLocation(new Validation.GCSPath("gs://bucket/tmp"))
Remove the invalid local/file URI and use a real GCS bucket that the pipeline's credentials can write to
If running on Dataflow, ensure the service account has storage.objects.create on the temp bucket

Example fix

// before
.apply("WriteBigQuery", BigQueryIO.writeTableRows().to(table)
    .withCustomGcsTempLocation(new Validation.GCSPath("/tmp/beam")));
// after
.apply("WriteBigQuery", BigQueryIO.writeTableRows().to(table)
    .withCustomGcsTempLocation(new Validation.GCSPath("gs://my-bucket/tmp")));

Defensive patterns

Strategy: validation

Validate before calling

// Java
String tempLocation = options.getTempLocation();
if (tempLocation == null || !tempLocation.startsWith("gs://")) {
  throw new IllegalArgumentException("--tempLocation must be a gs:// path, got: " + tempLocation);
}

Type guard

// Java
boolean isGcs = tempLocation != null && tempLocation.startsWith("gs://");

Try / catch

// Java
try {
  GcsPath.fromUri(tempLocation);
} catch (IllegalArgumentException e) {
  throw new IllegalArgumentException("Set --tempLocation to a gs:// URI, got: " + tempLocation, e);
}

Prevention

Always pass --tempLocation=gs://<bucket>/<path> when running pipelines with BigQuery sinks
Prefer withCustomGcsTempLocation() with an explicit GCS path in code
Ensure the runner's default temp location isn't a local directory
Grant the pipeline service account write access to the temp bucket

When it happens

Trigger: Calling BigQueryIO.write() with withCustomGcsTempLocation("/local/path") or a non-GCS URI (http://, s3://, plain path), especially with custom bigQueryServices

Common situations: Setting --tempLocation to a non-GCS value (local dir or another cloud's URI) while running with BigQuery Storage API or custom services; forgetting to set tempLocation at all in test harnesses using testBigQueryServices.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5dee4ed3be5ff1c1. Report an issue: GitHub.