apache/beam · error · ValueError

Row missing %r column. Row keys

Error message

Row missing %r column. Row keys: %s

What it means

While streaming rows read from a BigQuery change-history stream, each row must contain the configured change timestamp column. If row.get(change_timestamp_column) returns None — the column is absent or its value is null — ValueError is raised, reporting the column name and the row's actual keys to help diagnose the mismatch.

Solutions

1. Check the row keys in the error message and correct the change_timestamp_column / query to include the actual timestamp column name.
2. Ensure the underlying change-history query selects the required metadata columns (_CHANGE_TIMESTAMP for CHANGES, _COMMIT_TIMESTAMP-related metadata for APPENDS).
3. Filter or exclude rows with NULL timestamp values upstream before this DoFn if nulls are expected in your table.

Example fix

// before
ReadFromBigQueryChangeHistory(query="SELECT data FROM `proj.ds.table`", ...)
// after
ReadFromBigQueryChangeHistory(query="SELECT data, _CHANGE_TIMESTAMP FROM `proj.ds.table`", ...)

if change_timestamp_column not in (row.keys() if isinstance(row, dict) else row_schema_fields):
    raise KeyError(f"query must select {change_timestamp_column}")

try:
    _ = row[change_timestamp_column]
except (KeyError, ValueError) as e:
    log.error("row missing change timestamp column: %s", e)

When it happens

Trigger: The configured _change_timestamp_column (e.g. '_CHANGE_TIMESTAMP' or a custom metadata column) is missing from rows returned by _read_stream, typically because the query used for the stream did not select that metadata column, or a row legitimately has NULL in that column.

Common situations: Custom row filters or a restricted column projection excluding the change timestamp column; using the wrong column name for the table (custom change-history columns differ per table); rows written with NULL timestamps by the producing pipeline.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• A BigQuery table or a query must be specified
• Bigquery dependencies are not installed.
• Bigquery dependencies are not installed.
• BigQuery source must be split before being read
• BigQuery storage source must be split before being read

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/6c2813a2ee525f30. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_change_history.py:886

        break

      stream_name = stream_names[i]
      _LOGGER.info(
          '[Read] try_claim(%d) succeeded: reading stream %s', i, stream_name)

      stream_rows = 0
      if self._emit_raw_batches:
        stream_batches = 0
        for raw_batch in self._read_stream_raw(stream_name):
          yield TimestampedValue(raw_batch, element.range_start)
          stream_batches += 1
        Metrics.counter('BigQueryChangeHistory',
                        'batches_emitted').inc(stream_batches)
      else:
        for row in self._read_stream(stream_name):
          ts = row.get(self._change_timestamp_column)
          if ts is None:
            raise ValueError(
                'Row missing %r column. Row keys: %s' %
                (self._change_timestamp_column, list(row.keys())))
          if isinstance(ts, datetime.datetime):
            ts = Timestamp.from_utc_datetime(ts)

          yield TimestampedValue(row, ts)
          stream_rows += 1
        Metrics.counter('BigQueryChangeHistory',
                        'rows_emitted').inc(stream_rows)

      streams_read += 1
      _LOGGER.info(
          '[Read] Finished reading stream %d for %s: %d rows',
          i,
          table_key,
          stream_rows)
      Metrics.counter('BigQueryChangeHistory', 'streams_read').inc()

View on GitHub (pinned to 12126d8942)