What causes "Query job %s failed, status: %s" in apache/beam?

Inspect the pretty-printed status in the message (and job.getStatus().getErrors()) for the underlying BigQuery error reason

apache/beam · error · IOException

Query job %s failed, status: %s

Error message

Query job %s failed, status: %s

What it means

BigQueryIo throws this IOException in BigQueryQueryHelper.executeQuery when a submitted BigQuery query job does not reach SUCCEEDED status after polling. The job was successfully started via JobService.startQueryJob, but BigQueryHelpers.parseStatus reported a non-successful terminal state (FAILED, UNKNOWN, etc.). The message includes the query job id and a pretty-printed job status with error details.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQueryHelper.java:200

      JobConfigurationQuery queryConfiguration =
          createBasicQueryConfig(query, flattenResults, useLegacySql)
              .setAllowLargeResults(true)
              .setDestinationTable(queryResultTable)
              .setCreateDisposition("CREATE_IF_NEEDED")
              .setWriteDisposition("WRITE_TRUNCATE")
              .setPriority(priority.name());

      if (kmsKey != null) {
        queryConfiguration.setDestinationEncryptionConfiguration(
            new EncryptionConfiguration().setKmsKeyName(kmsKey));
      }

      JobService jobService = bqServices.getJobService(options);
      jobService.startQueryJob(jobReference, queryConfiguration);
      Job job = jobService.pollJob(jobReference, JOB_POLL_MAX_RETRIES);
      if (BigQueryHelpers.parseStatus(job) != Status.SUCCEEDED) {
        throw new IOException(
            String.format(
                "Query job %s failed, status: %s",
                queryJobId, BigQueryHelpers.statusToPrettyString(job.getStatus())));
      }

      LOG.info("Query job {} completed", queryJobId);
      return queryResultTable;

    } catch (RuntimeException | IOException | InterruptedException e) {
      throw e;
    } catch (Exception e) {
      throw new RuntimeException(e);
    }
  }

  private static JobConfigurationQuery createBasicQueryConfig(
      String query, Boolean flattenResults, Boolean useLegacySql) {
    return new JobConfigurationQuery()

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the pretty-printed status in the message (and job.getStatus().getErrors()) for the underlying BigQuery error reason
Run the query text manually in the BigQuery console to reproduce and fix SQL syntax or reference errors
Verify the referenced datasets/tables exist and the service account has bigquery.jobs.create and read permissions
Check project quota/usage if the status indicates rate or resource limits
Confirm the SQL dialect matches the useLegacySql configuration

Example fix

// before: silently failing query with wrong dialect
BigQueryIO.readTableRows().fromQuery("SELECT * FROM my_dataset.my_table")
// after: validate the query runs before the pipeline, and set dialect explicitly
BQ QueryUtils.validateQuery("SELECT * FROM `my-project.my_dataset.my_table`", /*useLegacySql=*/ false);
BigQueryIO.readTableRows().fromQuery("SELECT * FROM `my-project.my_dataset.my_table`").withMethod(Method.QUERY)

Defensive patterns

Strategy: try-catch

Validate before calling

// Validate the query before running the pipeline (dry run)
QueryRequest dryRun = QueryRequest.newBuilder(query).setUseLegacySql(false).setDryRun(true).build();
dataset.query(dryRun); // throws early with the real BigQuery validation error

Type guard

// Java: check job status before relying on results
boolean jobSucceeded(com.google.api.services.bigquery.model.Job job) {
  return job != null && job.getStatus() != null && "DONE".equals(job.getStatus().getState())
      && job.getStatus().getErrorResult() == null;
}

Try / catch

try {
  pipeline.apply(BigQueryIO.readTableRows().fromQuery(query));
} catch (IOException e) {
  if (e.getMessage().contains("Query job") && e.getMessage().contains("failed")) {
    LOG.error("BigQuery query job failed; check SQL, permissions and quotas", e);
    throw new ValidationException("Invalid BigQuery query: " + e.getCause(), e);
  }
  throw e;
}

Prevention

Dry-run every query in CI before deploying the pipeline
Use fully qualified standard-SQL table names (project.dataset.table) with useLegacySql=false
Verify service account permissions (bigquery.jobs.create, table read) per project
Monitor BigQuery quotas and slot usage for query-heavy pipelines

When it happens

Trigger: Executing a BigQueryIO query (e.g. BigQueryIO.readTableRows().fromQuery(...)) where the query is invalid (syntax error, unknown table/column), the job exceeds quotas, or the service returns a FAILED status during jobService.pollJob within JOB_POLL_MAX_RETRIES.

Common situations: Typo in SQL or dataset/table names; referencing tables the caller lacks permission to read; query exceeding the per-project concurrent quota or slot limits; using legacy SQL dialect while useLegacySql is false (or vice versa); job killed or failed server-side before completion.

Understand the failure class

Background: Database query failed: Internal Server Error 500s wrapping SQL, Prisma, and connection failures — what to check first — this error's family across 16 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c6aa4a02a36a613f. Report an issue: GitHub.