apache/beam · error · ImportError

pyarrow is required for ReadBigQueryChangeHistory. Install i

Error message

pyarrow is required for ReadBigQueryChangeHistory. Install it with: pip install pyarrow

What it means

ReadBigQueryChangeHistory also requires pyarrow to decode arrow RecordBatches returned by the Storage Read API. If the module-level pyarrow import is None, __init__ raises ImportError with the pip install command, since the arrow-batch read path cannot run without it.

Source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_change_history.py:1210

      buffer_sec: float = 10,
      project: Optional[str] = None,
      temp_dataset: Optional[str] = None,
      location: Optional[str] = None,
      change_type_column: str = 'change_type',
      change_timestamp_column: str = 'change_timestamp',
      columns: Optional[list[str]] = None,
      row_filter: Optional[str] = None,
      batch_arrow_read: bool = True,
      max_split_rounds: int = 1,
      reshuffle_decompress: bool = True) -> None:
    super().__init__()
    if bq_storage is None:
      raise ImportError(
          'google-cloud-bigquery-storage is required for '
          'ReadBigQueryChangeHistory. Install it with: '
          'pip install google-cloud-bigquery-storage')
    if pyarrow is None:
      raise ImportError(
          'pyarrow is required for ReadBigQueryChangeHistory. '
          'Install it with: pip install pyarrow')
    if change_function not in ('CHANGES', 'APPENDS'):
      raise ValueError(
          f"change_function must be 'CHANGES' or 'APPENDS', "
          f"got '{change_function}'")
    if poll_interval_sec < 15:
      raise ValueError(
          f'poll_interval_sec must be >= 15, got {poll_interval_sec}')
    if buffer_sec < 0:
      raise ValueError(f'buffer_sec must be >= 0, got {buffer_sec}')
    self._table = table
    self._poll_interval_sec = poll_interval_sec
    self._start_time = start_time
    self._stop_time = stop_time
    self._change_function = change_function
    self._buffer_sec = buffer_sec
    self._project = project

View on GitHub (pinned to 12126d8942)

Solutions

Run: pip install pyarrow
Install with GCP extras: pip install 'apache-beam[gcp]'
If the import fails due to binary incompatibility, reinstall a matching pyarrow/numpy pair: pip install --force-reinstall pyarrow numpy.

Example fix

// before
pip install apache-beam
// after
pip install 'apache-beam[gcp]'  # includes pyarrow

Defensive patterns

Strategy: fallback

Validate before calling

try:
    import pyarrow
except ImportError:
    raise SystemExit("pip install pyarrow")

Try / catch

try:
    transform = ReadFromBigQueryChangeHistory(...)
except ImportError as e:
    log.error("install missing dependency: %s", e)

Prevention

Add pyarrow to your requirements and pin a version compatible with your numpy
Use 'apache-beam[gcp]' which pulls pyarrow
If import fails due to binary conflicts, force-reinstall pyarrow and numpy together

When it happens

Trigger: Instantiating ReadFromBigQueryChangeHistory in an environment where pyarrow is not installed (pyarrow is None at import time), regardless of batch_arrow_read setting, because the transform validates both dependencies up front.

Common situations: Slim CI/deployment images without pyarrow; conflicting numpy/pyarrow wheels causing failed import; forgetting the gcp extras that pull in pyarrow.

Understand the failure class

Background: "X is not installed. Please install it with pip install Y": missing optional dependency errors — ImportError/ValueError raised when a library's optional extra was never installed — this error's family across 22 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/827afd275d55650b. Report an issue: GitHub.