apache/beam · error · RuntimeException

Error while converting table schema %s to JSON!

Error message

Error while converting table schema %s to JSON!

What it means

Thrown when serializing the BigQuery table schema to JSON (via Jackson JSON_FACTORY) fails with an IOException inside the Avro datum-reader setup for typed reads. Beam needs the schema as JSON to configure GenericDatumTransformer for converting Avro records.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:806

   *   }
   * }).from("...");
   * }</pre>
   */
  public static <T> TypedRead<T> read(SerializableFunction<SchemaAndRecord, T> parseFn) {
    return new AutoValue_BigQueryIO_TypedRead.Builder<T>()
        .setValidate(true)
        .setWithTemplateCompatibility(false)
        .setBigQueryServices(new BigQueryServicesImpl())
        .setDatumReaderFactory(
            (SerializableFunction<TableSchema, AvroSource.DatumReaderFactory<T>>)
                input -> {
                  try {
                    String jsonTableSchema = BigQueryIO.JSON_FACTORY.toString(input);
                    return (AvroSource.DatumReaderFactory<T>)
                        (writer, reader) ->
                            new GenericDatumTransformer<>(parseFn, jsonTableSchema, writer);
                  } catch (IOException e) {
                    throw new RuntimeException(
                        "Error while converting table schema " + input + " to JSON!", e);
                  }
                })
        // TODO: Remove setParseFn once https://github.com/apache/beam/issues/21076 is fixed.
        .setParseFn(parseFn)
        .setMethod(TypedRead.Method.DEFAULT)
        .setUseAvroLogicalTypes(false)
        .setFormat(DataFormat.AVRO)
        .setProjectionPushdownApplied(false)
        .setBadRecordErrorHandler(new DefaultErrorHandler<>())
        .setBadRecordRouter(BadRecordRouter.THROWING_ROUTER)
        .build();
  }

  /**
   * Reads from a BigQuery table or query and returns a {@link PCollection} with one element per
   * each row of the table or query result. This API directly deserializes BigQuery AVRO data to the
   * input class, based on the appropriate {@link org.apache.avro.io.DatumReader}.

View on GitHub (pinned to 12126d8942)

Solutions

1. Inspect the table schema for unusual/unsupported field types and simplify if possible
2. Resolve Jackson dependency conflicts (check the shaded beam-sdks-java-io-google-cloud-platform jar is used)
3. Update to a newer Beam version where schema serialization edge cases are fixed

try {
  String json = BigQueryIO.JSON_FACTORY.toString(tableSchema);
} catch (IOException e) {
  throw new IllegalStateException("Failed to serialize table schema to JSON: " + tableSchema, e);
}

When it happens

Trigger: BigQueryIO.read(...).withParseFn(...) using the DEFAULT/FILE_LOADS-style Avro path where the TableSchema cannot be serialized to a JSON string.

Common situations: Corrupt or unsupported schema fields coming from BigQuery; Jackson serialization issues in shaded/dependency-conflict environments.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Field type%s %s not supported when converting between JSON a
• Value "{value}" is out of range for the type of the field de
• Reserved field name <field.name()> in user schema.
• RECORD/STRUCT are not primitive types
• Unknown BigQuery type: " + bqType

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/e6108ad32a99c6d6. Report an issue: GitHub.