apache/beam · error · RuntimeError

BigQuery job failed. Error Result

Error message

BigQuery job {} failed. Error Result: {}

What it means

Raised by BigQueryWrapper.wait_for_bq_job when polling shows the BigQuery job reached state DONE with a non-empty status.errorResult — i.e. BigQuery itself reported the job failed (bad SQL, quota, missing table, permissions, etc.). Beam surfaces the server's errorResult verbatim as a RuntimeError.

Solutions

1. Read the errorResult message in the exception and fix the underlying BigQuery job error (SQL, schema, quota).
2. Validate the query with dry-run (client query with dry_run=True) before submitting.
3. Check the job in the BigQuery console / bq CLI ('bq show -j <job_id>') for detailed error stack.
4. Verify IAM permissions (bigquery.jobs.create, data read) and quota limits for the project/location.

Example fix

// before
client.query('SELCT * FROM ds.t')  # typo causes job failure

// after
client.query('SELECT * FROM ds.t')

# dry-run the query before submitting a real job
dry_run = client.query(sql, job_config=bigquery.QueryJobConfig(dry_run=True, use_query_cache=False))
print(f'query will process {dry_run.total_bytes_processed} bytes')

def job_succeeded(job):
    return job.status.state == 'DONE' and not job.status.errorResult

try:
    wrapper.wait_for_bq_job(job_ref, sleep_duration_sec=10)
except RuntimeError as e:
    if 'BigQuery job' in str(e) and 'failed' in str(e):
        handle_bq_failure(job_ref.jobId, str(e))  # inspect errorResult, alert, retry with fixed input
    else:
        raise

When it happens

Trigger: wait_for_bq_job(job_reference, ...) called by _execute_query or _export_files when the submitted query or export job fails server-side; inspect job.status.errorResult for the actual reason.

Common situations: Invalid SQL syntax in a query; source table missing or renamed; per-user quota or slot exhaustion; dataset-level IAM changes removing bigquery.jobs.create; load jobs failing schema mismatch.

Related errors

• A BigQuery table or a query must be specified
• AvroRowWriter is not readable
• Bigquery dependencies are not installed.
• Bigquery dependencies are not installed.
• BigQuery source must be split before being read

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/3bf42c2784b0f9cb. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:699

    Args:
      job_reference: bigquery.JobReference instance.
      sleep_duration_sec: Specifies the delay in seconds between retries.
      max_retries: The total number of times to retry. If equals to 0,
        the function waits forever.

    Raises:
      `RuntimeError`: If the job is FAILED or the number of retries has been
        reached.
    """
    retry = 0
    while True:
      retry += 1
      job = self.get_job(
          job_reference.projectId, job_reference.jobId, job_reference.location)
      _LOGGER.info('Job %s status: %s', job.id, job.status.state)
      if job.status.state == 'DONE' and job.status.errorResult:
        raise RuntimeError(
            'BigQuery job {} failed. Error Result: {}'.format(
                job_reference.jobId, job.status.errorResult))
      elif job.status.state == 'DONE':
        return True
      else:
        time.sleep(sleep_duration_sec)
        if max_retries != 0 and retry >= max_retries:
          raise RuntimeError('The maximum number of retries has been reached')

  @retry.with_exponential_backoff(
      num_retries=MAX_RETRIES,
      retry_filter=retry.retry_on_server_errors_and_timeout_filter)
  def _get_query_results(
      self,
      project_id,
      job_id,
      page_token=None,
      max_results=10000,

View on GitHub (pinned to 12126d8942)