apache/beam · error · BadRequest

Could not execute the query

Error message

Could not execute the query: {query}. Please check if the query is properly formatted and the BigQuery table exists. {e}

What it means

_execute_query() wraps google.api_core.exceptions.BadRequest raised while running the enrichment query in BigQuery, re-raising it with the formatted query text to help diagnose formatting errors or a missing table.

Solutions

1. Print/inspect the embedded query text and run it manually in the BigQuery console to find the syntax problem
2. Verify the table's fully qualified name (project.dataset.table) and that it exists
3. Check that the number/order of format placeholders in the template matches the values passed
4. Catch BadRequest around enrichment calls if empty/bad results are expected

Example fix

// before
query_template = 'SELECT * FROM `proj.ds.tbl` WHERE id = {}'
// after (quote and match placeholders)
query_template = "SELECT * FROM `proj.ds.tbl` WHERE id = '{}'"

def validate_template(template, n_args):
    if template.count('{}') != n_args:
        raise ValueError(f'Query template has {template.count("{}")} placeholders, expected {n_args}')

from google.api_core.exceptions import BadRequest
try:
    out = handler(row)
except BadRequest as e:
    logger.error('BigQuery enrichment query failed: %s', e)
    out = beam.Row()

When it happens

Trigger: __call__ executes a malformed SQL query (bad query_template placeholders/format args), or the query references a nonexistent table/dataset, and BigQuery returns BadRequest.

Common situations: query_template placeholders not matching the number of format values; typos in table or column names; dataset deleted or in a different project.

Understand the failure class

Background: "query failed", "%w: SQL error" — wrapped database query errors in Go libraries explained — this error's family across 3 libraries.

Related errors

• A BigQuery table or a query must be specified
• A function must be provided to convert the input type into…
• A schema is required in order to prepare rows for writing…
• A schema must be provided when writing to BigQuery using…
• Beam SQL cannot convert Timestamp values with…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c41dc60c67257534. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/enrichment_handlers/bigquery.py:182

      if max_batch_duration_secs is not None:
        self._batching_kwargs[
            'max_batch_duration_secs'] = max_batch_duration_secs

  def __enter__(self):
    self.client = bigquery.Client(project=self.project, **self.kwargs)

  def _execute_query(self, query: str):
    try:
      results = self.client.query(query=query).result()
      row_list = [dict(row.items()) for row in results]
      if not row_list:
        return None
      if self._batching_kwargs:
        return row_list
      else:
        return row_list[0]
    except BadRequest as e:
      raise BadRequest(
          f'Could not execute the query: {query}. Please check if '
          f'the query is properly formatted and the BigQuery '
          f'table exists. {e}')
    except RuntimeError as e:
      raise RuntimeError(f"Could not complete the query request: {query}. {e}")

  def create_row_key(self, row: beam.Row):
    if self.condition_value_fn:
      return tuple(self.condition_value_fn(row))
    if self.fields:
      row_dict = row._asdict()
      return (tuple(row_dict[field] for field in self.fields))
    raise ValueError("Either fields or condition_value_fn must be specified")

  def __call__(self, request: Union[beam.Row, list[beam.Row]], *args, **kwargs):
    if isinstance(request, list):
      values = []
      responses = []

View on GitHub (pinned to 12126d8942)