{"record":{"id":"1603312d3529af8c","repo":"apache/beam","slug":"a-schema-is-required-to-write-non-schema-d-data","errorCode":null,"errorMessage":"A schema is required to write non-schema'd data.","messagePattern":"A schema is required to write non-schema'd data\\.","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"sdks/python/apache_beam/io/parquetio.py","lineNumber":623,"sourceCode":"          mime_type,\n          triggering_frequency\n      )\n\n  def expand(self, pcoll):\n    if (not pcoll.is_bounded and self._sink.shard_name_template\n        == filebasedsink.DEFAULT_SHARD_NAME_TEMPLATE):\n      self._sink.shard_name_template = (\n          filebasedsink.DEFAULT_WINDOW_SHARD_NAME_TEMPLATE)\n      self._sink.shard_name_format = self._sink._template_to_format(\n          self._sink.shard_name_template)\n      self._sink.shard_name_glob_format = self._sink._template_to_glob_format(\n          self._sink.shard_name_template)\n\n    if self._schema is None:\n      try:\n        beam_schema = schemas.schema_from_element_type(pcoll.element_type)\n      except TypeError as exn:\n        raise ValueError(\n            \"A schema is required to write non-schema'd data.\") from exn\n      self._sink._schema = (\n          arrow_type_compatibility.arrow_schema_from_beam_schema(beam_schema))\n      convert_fn = _BeamRowsToArrowTable()\n    else:\n      convert_fn = _RowDictionariesToArrowTable(\n          self._schema, self._row_group_buffer_size, self._record_batch_size)\n    if pcoll.is_bounded:\n      return pcoll | ParDo(convert_fn) | Write(self._sink)\n    else:\n      self._sink.convert_fn = convert_fn\n      return pcoll | Write(self._sink)\n\n  def display_data(self):\n    return {\n        'sink_dd': self._sink,\n        'row_group_buffer_size': str(self._row_group_buffer_size)\n    }","sourceCodeStart":605,"sourceCodeEnd":641,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/python/apache_beam/io/parquetio.py#L605-L641","documentation":"_WriteToParquet.expand needs a schema to know how to convert incoming rows into Arrow tables. If the sink was constructed without a `schema` and the input PCollection's element type is not a Beam schema'd row class (so schema_from_element_type fails), the writer cannot proceed and raises this ValueError.","triggerScenarios":"beam.io.parquetio.WriteToParquet(file_path_prefix) without a `schema=` argument, writing a PCollection of plain dicts or other non-schema'd types. If the element type is a schema'd row class, the schema is inferred instead and no error occurs.","commonSituations":"Writing dict rows after migrating code that previously passed a schema; forgetting schema= after refactoring the input element type away from a NamedTuple/Beam Row.","solutions":["Pass schema= to WriteToParquet, e.g. WriteToParquet(path, schema=pa.schema([('name', pa.string()), ('age', pa.int64())])).","Convert input to a Beam schema'd row class (beam.Row or NamedTuple registered via @dataclass with beam type hints) so the schema can be inferred.","Define the schema from the data with beam.Row(...) type hints on the PCollection."],"exampleFix":"// before\nresult | WriteToParquet('out.parquet')  # dict rows, no schema\n// after\nresult | WriteToParquet('out.parquet', schema=pa.schema([('name', pa.string()), ('age', pa.int64())]))","handlingStrategy":"validation","validationCode":"if schema is None and not hasattr(element_type, 'beam_schema'):\n    schema = pa.schema([('col1', pa.string())])  # derive or define explicitly","typeGuard":"def has_beam_schema(element_type) -> bool:\n    try:\n        schemas.schema_from_element_type(element_type)\n        return True\n    except TypeError:\n        return False","tryCatchPattern":"try:\n    result |= WriteToParquet(path, schema=schema)\nexcept ValueError as e:\n    if \"schema is required\" in str(e):\n        result |= WriteToParquet(path, schema=derive_schema(pcoll))","preventionTips":["Always pass an explicit schema= to WriteToParquet when writing dicts.","Or ensure the PCollection element type is a Beam-schema'd row class.","Check with apache_beam.typehints that the input type carries a schema."],"tags":["python","apache-beam","io","parquet","schema"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}