apache/beam · error · ValueError

Due to ARROW-9424, writing with LZ4 compression is not…

Error message

Due to ARROW-9424, writing with LZ4 compression is not supported in pyarrow 1.x, please use a different pyarrow version or a different codec. Your pyarrow version: {pa.__version__}

What it means

WriteToParquet checks the installed pyarrow major version: pyarrow 1.x has a bug (ARROW-9424) that corrupts or fails writes when LZ4 compression is requested. To prevent bad output files, the sink refuses construction when codec.lower() == 'lz4' with pyarrow 1.x installed.

Solutions

Upgrade pyarrow to >= 2.x (pip install 'pyarrow>=2').
Use a different codec such as 'snappy', 'gzip', 'zstd', or 'brotli'.
Check pyarrow.__version__ in your environment and align with Beam's supported range.

Example fix

// before
WriteToParquet('out', schema=s, codec='lz4')  # pyarrow 1.x
// after
WriteToParquet('out', schema=s, codec='snappy')  # or upgrade pyarrow>=2

Defensive patterns

Strategy: validation

Validate before calling

import pyarrow as pa
if int(pa.__version__.split('.')[0]) == 1 and codec.lower() == 'lz4':
    codec = 'snappy'

Type guard

def codec_supported(codec: str, pa_version: str) -> bool:
    major = int(pa_version.split('.')[0])
    return not (major == 1 and codec.lower() == 'lz4')

Try / catch

try:
    sink = WriteToParquet(path, schema=s, codec='lz4')
except ValueError as e:
    if 'ARROW-9424' in str(e):
        sink = WriteToParquet(path, schema=s, codec='snappy')

Prevention

Pin pyarrow>=2 in requirements.
Run a version check at pipeline startup.
Prefer snappy/gzip codecs for portability.

When it happens

Trigger: Constructing WriteToParquet(..., codec='lz4') (or 'LZ4') while pyarrow 1.x is installed.

Common situations: Pinned old pyarrow 1.x in requirements from an old project; upgrading Beam but not pyarrow; copying a code sample using LZ4 codec.

Understand the failure class

Background: "is not a compatible type" / "cannot merge" errors: when a value's type doesn't match what the library requires — this error's family across 65 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/245bfe1dcf2ca13d. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/parquetio.py:823

      num_shards,
      shard_name_template,
      mime_type,
      triggering_frequency):
    super().__init__(
        file_path_prefix,
        file_name_suffix=file_name_suffix,
        num_shards=num_shards,
        shard_name_template=shard_name_template,
        coder=None,
        mime_type=mime_type,
        # Compression happens at the block level using the supplied codec, and
        # not at the file level.
        compression_type=CompressionTypes.UNCOMPRESSED,
        triggering_frequency=triggering_frequency)
    self._schema = schema
    self._codec = codec
    if ARROW_MAJOR_VERSION == 1 and self._codec.lower() == "lz4":
      raise ValueError(
          "Due to ARROW-9424, writing with LZ4 compression is not supported in "
          "pyarrow 1.x, please use a different pyarrow version or a different "
          f"codec. Your pyarrow version: {pa.__version__}")
    self._use_deprecated_int96_timestamps = use_deprecated_int96_timestamps
    if use_compliant_nested_type and ARROW_MAJOR_VERSION < 4:
      raise ValueError(
          "With ARROW-11497, use_compliant_nested_type is only supported in "
          "pyarrow version >= 4.x, please use a different pyarrow version. "
          f"Your pyarrow version: {pa.__version__}")
    self._use_compliant_nested_type = use_compliant_nested_type
    self._file_handle = None

  def open(self, temp_path):
    self._file_handle = super().open(temp_path)
    if ARROW_MAJOR_VERSION < 4:
      return pq.ParquetWriter(
          self._file_handle,
          self._schema,

View on GitHub (pinned to 12126d8942)