apache/beam · error · ValueError

Either where_clause_fields or where_clause_value_fn must be…

Error message

Either where_clause_fields or where_clause_value_fn must be specified

What it means

create_row_key builds a dedup/batch key from each row. For TableFieldsQueryConfig it tuples the where_clause_fields values; for any other config type without where_clause_fields it raises ValueError saying where_clause_fields or where_clause_value_fn must be specified.

Solutions

1. Use TableFieldsQueryConfig with where_clause_fields set for batched requests.
2. Disable batching and use single-request processing for CustomQueryConfig.
3. Ensure the passed config type supports row-key generation.

Example fix

# before
handler = CloudSQLEnrichmentHandler(query_config=CustomQueryConfig(query=...), batch_lookup=True, ...)
# after
handler = CloudSQLEnrichmentHandler(query_config=TableFieldsQueryConfig(table_name="users", table_fields=["id"], where_clause_template="id = {id}", where_clause_fields=["id"]), batch_lookup=True, ...)

When it happens

Trigger: Batched enrichment (_process_batch_request calls create_row_key) with a query config that is not TableFieldsQueryConfig and provides no where_clause_fields / where_clause_value_fn, e.g. CustomQueryConfig.

Common situations: Using CustomQueryConfig with batched lookups; forgetting to populate where_clause_fields; switching configs without updating batch settings.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

• Batch size must be a positive integer
• Caching is not supported for CustomQueryConfig. Consider…
• Database host cannot be empty
• Instance connection URI cannot be empty
• Parameter binding not supported for

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/05c792e36d8801d1. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/transforms/enrichment_handlers/cloudsql.py:610

    Args:
      template: SQL template string with named parameters (e.g., "id = :id")

    Returns:
      List of parameter names found in the template (e.g., ["id"])
    """
    return re.findall(r':(\w+)', template)

  def create_row_key(self, row: beam.Row):
    if isinstance(self._query_config, TableFunctionQueryConfig):
      return tuple(self._query_config.where_clause_value_fn(row))
    if isinstance(self._query_config, TableFieldsQueryConfig):
      row_dict = row._asdict()
      return (
          tuple(
              row_dict[where_clause_field]
              for where_clause_field in self._query_config.where_clause_fields))
    raise ValueError(
        "Either where_clause_fields or where_clause_value_fn must be specified")

  def get_cache_key(self, request: Union[beam.Row, list[beam.Row]]):
    if isinstance(self._query_config, CustomQueryConfig):
      raise NotImplementedError(
          "Caching is not supported for CustomQueryConfig. "
          "Consider using TableFieldsQueryConfig or " +
          "TableFunctionQueryConfig instead.")

    if isinstance(request, list):
      cache_keys = []
      for req in request:
        req_dict = req._asdict()
        try:
          if isinstance(self._query_config, TableFunctionQueryConfig):
            current_values = self._query_config.where_clause_value_fn(req)
          elif isinstance(self._query_config, TableFieldsQueryConfig):
            current_values = [

View on GitHub (pinned to 12126d8942)