{"record":{"id":"3158a0aa8df197d1","repo":"apache/beam","slug":"with-arrow-11497-use-compliant-nested-type-is-only-supported","errorCode":null,"errorMessage":"With ARROW-11497, use_compliant_nested_type is only supported in pyarrow version >= 4.x, please use a different pyarrow version. Your pyarrow version: {pa.__version__}","messagePattern":"With ARROW-11497, use_compliant_nested_type is only supported in pyarrow version >= 4\\.x, please use a different pyarrow version\\. Your pyarrow version: (.+?)","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/parquetio.py","lineNumber":829,"sourceCode":"        file_name_suffix=file_name_suffix,\n        num_shards=num_shards,\n        shard_name_template=shard_name_template,\n        coder=None,\n        mime_type=mime_type,\n        # Compression happens at the block level using the supplied codec, and\n        # not at the file level.\n        compression_type=CompressionTypes.UNCOMPRESSED,\n        triggering_frequency=triggering_frequency)\n    self._schema = schema\n    self._codec = codec\n    if ARROW_MAJOR_VERSION == 1 and self._codec.lower() == \"lz4\":\n      raise ValueError(\n          \"Due to ARROW-9424, writing with LZ4 compression is not supported in \"\n          \"pyarrow 1.x, please use a different pyarrow version or a different \"\n          f\"codec. Your pyarrow version: {pa.__version__}\")\n    self._use_deprecated_int96_timestamps = use_deprecated_int96_timestamps\n    if use_compliant_nested_type and ARROW_MAJOR_VERSION < 4:\n      raise ValueError(\n          \"With ARROW-11497, use_compliant_nested_type is only supported in \"\n          \"pyarrow version >= 4.x, please use a different pyarrow version. \"\n          f\"Your pyarrow version: {pa.__version__}\")\n    self._use_compliant_nested_type = use_compliant_nested_type\n    self._file_handle = None\n\n  def open(self, temp_path):\n    self._file_handle = super().open(temp_path)\n    if ARROW_MAJOR_VERSION < 4:\n      return pq.ParquetWriter(\n          self._file_handle,\n          self._schema,\n          compression=self._codec,\n          use_deprecated_int96_timestamps=self._use_deprecated_int96_timestamps)\n    return pq.ParquetWriter(\n        self._file_handle,\n        self._schema,\n        compression=self._codec,","sourceCodeStart":811,"sourceCodeEnd":847,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/parquetio.py#L811-L847","documentation":"WriteToParquet's `use_compliant_nested_type` option relies on pyarrow fixes from ARROW-11497, which are only present in pyarrow >= 4.x. Requesting compliant nested types with an older pyarrow raises this ValueError at sink construction to avoid writing incompatible/incorrect nested Parquet structures.","triggerScenarios":"WriteToParquet(..., use_compliant_nested_type=True) with pyarrow major version < 4 installed (default value of this flag is also True on newer Beam).","commonSituations":"Legacy environment pinned to pyarrow 1.x-3.x; Docker image with old pyarrow; upgrading Beam but leaving pyarrow pinned.","solutions":["Upgrade pyarrow to >= 4.x (pip install 'pyarrow>=4').","Pass use_compliant_nested_type=False if you must stay on old pyarrow and accept legacy nested layout.","Rebuild your environment/requirements to include a modern pyarrow compatible with your Beam version."],"exampleFix":"// before\nWriteToParquet('out', schema=s, use_compliant_nested_type=True)  # pyarrow 3.x\n// after\nWriteToParquet('out', schema=s, use_compliant_nested_type=False)  # or upgrade pyarrow>=4","handlingStrategy":"validation","validationCode":"import pyarrow as pa\nif int(pa.__version__.split('.')[0]) < 4 and use_compliant_nested_type:\n    use_compliant_nested_type = False","typeGuard":"def supports_compliant_nested_type(pa_version: str) -> bool:\n    return int(pa_version.split('.')[0]) >= 4","tryCatchPattern":"try:\n    sink = WriteToParquet(path, schema=s, use_compliant_nested_type=True)\nexcept ValueError as e:\n    if 'ARROW-11497' in str(e):\n        sink = WriteToParquet(path, schema=s, use_compliant_nested_type=False)","preventionTips":["Pin pyarrow>=4 in requirements.","Only enable use_compliant_nested_type when the runtime pyarrow supports it.","Test Parquet sinks in CI against the same pyarrow version as production."],"tags":["python","apache-beam","parquet","pyarrow","version-compatibility"],"backgroundTag":"incompatible-source-type","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-20T03:17:13.778Z"}