{"record":{"id":"245bfe1dcf2ca13d","repo":"apache/beam","slug":"due-to-arrow-9424-writing-with-lz4-compression-is-not","errorCode":null,"errorMessage":"Due to ARROW-9424, writing with LZ4 compression is not supported in pyarrow 1.x, please use a different pyarrow version or a different codec. Your pyarrow version: {pa.__version__}","messagePattern":"Due to ARROW-9424, writing with LZ4 compression is not supported in pyarrow 1\\.x, please use a different pyarrow version or a different codec\\. Your pyarrow version: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/parquetio.py","lineNumber":823,"sourceCode":"      num_shards,\n      shard_name_template,\n      mime_type,\n      triggering_frequency):\n    super().__init__(\n        file_path_prefix,\n        file_name_suffix=file_name_suffix,\n        num_shards=num_shards,\n        shard_name_template=shard_name_template,\n        coder=None,\n        mime_type=mime_type,\n        # Compression happens at the block level using the supplied codec, and\n        # not at the file level.\n        compression_type=CompressionTypes.UNCOMPRESSED,\n        triggering_frequency=triggering_frequency)\n    self._schema = schema\n    self._codec = codec\n    if ARROW_MAJOR_VERSION == 1 and self._codec.lower() == \"lz4\":\n      raise ValueError(\n          \"Due to ARROW-9424, writing with LZ4 compression is not supported in \"\n          \"pyarrow 1.x, please use a different pyarrow version or a different \"\n          f\"codec. Your pyarrow version: {pa.__version__}\")\n    self._use_deprecated_int96_timestamps = use_deprecated_int96_timestamps\n    if use_compliant_nested_type and ARROW_MAJOR_VERSION < 4:\n      raise ValueError(\n          \"With ARROW-11497, use_compliant_nested_type is only supported in \"\n          \"pyarrow version >= 4.x, please use a different pyarrow version. \"\n          f\"Your pyarrow version: {pa.__version__}\")\n    self._use_compliant_nested_type = use_compliant_nested_type\n    self._file_handle = None\n\n  def open(self, temp_path):\n    self._file_handle = super().open(temp_path)\n    if ARROW_MAJOR_VERSION < 4:\n      return pq.ParquetWriter(\n          self._file_handle,\n          self._schema,","sourceCodeStart":805,"sourceCodeEnd":841,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/parquetio.py#L805-L841","documentation":"WriteToParquet checks the installed pyarrow major version: pyarrow 1.x has a bug (ARROW-9424) that corrupts or fails writes when LZ4 compression is requested. To prevent bad output files, the sink refuses construction when codec.lower() == 'lz4' with pyarrow 1.x installed.","triggerScenarios":"Constructing WriteToParquet(..., codec='lz4') (or 'LZ4') while pyarrow 1.x is installed.","commonSituations":"Pinned old pyarrow 1.x in requirements from an old project; upgrading Beam but not pyarrow; copying a code sample using LZ4 codec.","solutions":["Upgrade pyarrow to >= 2.x (pip install 'pyarrow>=2').","Use a different codec such as 'snappy', 'gzip', 'zstd', or 'brotli'.","Check pyarrow.__version__ in your environment and align with Beam's supported range."],"exampleFix":"// before\nWriteToParquet('out', schema=s, codec='lz4')  # pyarrow 1.x\n// after\nWriteToParquet('out', schema=s, codec='snappy')  # or upgrade pyarrow>=2","handlingStrategy":"validation","validationCode":"import pyarrow as pa\nif int(pa.__version__.split('.')[0]) == 1 and codec.lower() == 'lz4':\n    codec = 'snappy'","typeGuard":"def codec_supported(codec: str, pa_version: str) -> bool:\n    major = int(pa_version.split('.')[0])\n    return not (major == 1 and codec.lower() == 'lz4')","tryCatchPattern":"try:\n    sink = WriteToParquet(path, schema=s, codec='lz4')\nexcept ValueError as e:\n    if 'ARROW-9424' in str(e):\n        sink = WriteToParquet(path, schema=s, codec='snappy')","preventionTips":["Pin pyarrow>=2 in requirements.","Run a version check at pipeline startup.","Prefer snappy/gzip codecs for portability."],"tags":["python","apache-beam","parquet","pyarrow","version-compatibility","compression"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}