apache/beam · error · IOException

Extract job failed, status: .

Error message

Extract job %s failed, status: %s.

What it means

BigQuerySourceBase.executeExtract launches a BigQuery extract (export) job to dump the table to GCS files, then checks the job status. If BigQueryHelpers.parseStatus reports anything other than SUCCEEDED, the job's status is wrapped in an IOException and rethrown. This means the server-side export job itself failed (or has a failed/error state), not the client call.

Solutions

  1. Read the pretty-printed job status in the message to get the exact BigQuery error (e.g. accessDenied, invalid, rateLimitExceeded) and fix that underlying cause first
  2. Verify the --tempLocation / tempWriteGcsBucket GCS bucket exists and the project's BigQuery service account has OBJECT_WRITER permission on it
  3. Confirm the source is a normal (non-external) table; external tables cannot be exported
  4. Retry the pipeline if the status indicates a transient BigQuery error; consider enabling BigQuery job retry options

Example fix

// before
pipeline.apply("Read", BigQueryIO.readTableRows().from("proj:ds.tbl").withMethod(EXTRACT));
// after
// ensure a valid, writable temp bucket and non-external table:
BigQueryIO.readTableRows()
    .from("proj:ds.tbl")
    .withMethod(Method.EXTRACT)
    .withTempWriteGcsBucket("my-writable-bucket") // must exist & allow BigQuery export
    .withKmsKey(...) // only if required by bucket policy
Defensive patterns

Strategy: try-catch

Validate before calling

// Pre-check: bucket exists & service account can write
Storage storage = StorageOptions.getDefaultInstance().getService();
if (storage.get(tempBucket) == null) throw new IllegalArgumentException("tempLocation bucket missing");
// and confirm the table is not external:
// bq show --format=json project:dataset.table | jq -r '.type' -> must be TABLE/VIEW exportable

Type guard

boolean isExportableTable(Table t) {
  return t != null && !"EXTERNAL".equals(t.getType());
}

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().startsWith("Extract job ")) {
    // parse BigQueryHelpers.statusToPrettyString payload for error.status and handle specific causes
  }
  throw e;
}

Prevention

When it happens

Trigger: Calling BigQueryIO.read/transform that extracts a table to temporary GCS files when the extract job ends in a non-SUCCEEDED state (quota exceeded, destination bucket missing/permission-denied, unsupported source such as an external or materialized-view-like table, invalid destination format/URI, transient BigQuery backend error).

Common situations: Exporting a table whose data exceeds the 1GB-per-file or multi-wildcard limits; GCS temp location bucket doesn't exist or the service account lacks write permission; attempting to export external tables; BigQuery transient job failures during heavy load.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/24400d47f57b4054. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceBase.java:238

            .setUseAvroLogicalTypes(useAvroLogicalTypes)
            .setDestinationUris(ImmutableList.of(destinationUri));

    Job extractJob;
    try {
      LOG.info("Starting BigQuery extract job: {}", jobId);
      jobService.startExtractJob(jobRef, extract);
      extractJob = jobService.pollJob(jobRef, JOB_POLL_MAX_RETRIES);
    } catch (IOException exn) {
      // The error messages thrown in this case are generic and misleading, so leave this breadcrumb
      // in case it's the root cause.
      LOG.warn(
          "Error extracting table. Note that external tables cannot be exported: "
              + "https://cloud.google.com/bigquery/docs/external-tables#external_table_limitations",
          exn);
      throw exn;
    }
    if (BigQueryHelpers.parseStatus(extractJob) != Status.SUCCEEDED) {
      throw new IOException(
          String.format(
              "Extract job %s failed, status: %s.",
              extractJob.getJobReference().getJobId(),
              BigQueryHelpers.statusToPrettyString(extractJob.getStatus())));
    }

    LOG.info("BigQuery extract job completed: {}", jobId);

    return BigQueryIO.getExtractFilePaths(extractDestinationDir, extractJob);
  }

  List<BoundedSource<T>> createSources(
      List<ResourceId> files, TableSchema schema, @Nullable List<MatchResult.Metadata> metadata)
      throws IOException, InterruptedException {
    String avroSchema = BigQueryAvroUtils.toGenericAvroSchema(schema).toString();

    AvroSource.DatumReaderFactory<T> factory = readerFactory.apply(schema);

View on GitHub (pinned to 12126d8942)