apache/beam · error · IllegalArgumentException

A function must be provided to convert the input type into a

Error message

A function must be provided to convert the input type into a TableRow or GenericRecord. Use BigQueryIO.Write.withFormatFunction or BigQueryIO.Write.withAvroFormatFunction to provide a formatting function. A format function is not required if Beam schemas are used.

What it means

BigQueryIO.Write requires a way to convert each pipeline element into a TableRow (or a GenericRecord for Avro/file loads). If the input type is not a TableRow itself and neither a format function nor an Avro format function is configured (and Beam schemas are not inferred), the write expansion throws IllegalArgumentException telling the developer to provide one.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java:4123

              formatFunction == null,
              "Only one of withFormatFunction or withAvroFormatFunction/withAvroWriter maybe set,"
                  + " not both.");

          SerializableFunction<@Nullable TableSchema, org.apache.avro.Schema> avroSchemaFactory =
              getAvroSchemaFactory();
          if (avroSchemaFactory == null) {
            checkArgument(
                hasSchema,
                "A schema must be provided if an avroFormatFunction "
                    + "is set but no avroSchemaFactory is defined.");
            avroSchemaFactory = DEFAULT_AVRO_SCHEMA_FACTORY;
          }
          rowWriterFactory = avroRowWriterFactory.prepare(dynamicDestinations, avroSchemaFactory);
        } else if (formatFunction != null) {
          rowWriterFactory =
              RowWriterFactory.tableRows(formatFunction, formatRecordOnFailureFunction);
        } else {
          throw new IllegalArgumentException(
              "A function must be provided to convert the input type into a TableRow or "
                  + "GenericRecord. Use BigQueryIO.Write.withFormatFunction or "
                  + "BigQueryIO.Write.withAvroFormatFunction to provide a formatting function. "
                  + "A format function is not required if Beam schemas are used.");
        }
      } else {
        checkArgument(
            avroRowWriterFactory == null,
            "When using a formatFunction, the AvroRowWriterFactory should be null");
        rowWriterFactory =
            RowWriterFactory.tableRows(
                checkArgumentNotNull(
                    formatFunction,
                    "A function must be provided to convert the input type into a TableRow or "
                        + "GenericRecord. Use BigQueryIO.Write.withFormatFunction or "
                        + "BigQueryIO.Write.withAvroFormatFunction to provide a formatting "
                        + "function. A format function is not required if Beam schemas are "
                        + "used."),

View on GitHub (pinned to 12126d8942)

Solutions

1. Add .withFormatFunction(myObj -> new TableRow().set(...)) to the BigQueryIO.Write transform.
2. For file/store loads of Avro records, use .withAvroFormatFunction(record -> ...) instead.
3. Alternatively define a Beam Schema for the input type (via @DefaultSchema / SchemaCoder) so a format function is not required.
4. If the input is already TableRow, verify the generic type parameter is TableRow; otherwise Beam cannot see it as such.

Example fix

// before
rows.apply("Write", BigQueryIO.<MyEvent>write().to("proj:ds.table"));
// after
rows.apply("Write", BigQueryIO.<MyEvent>write().to("proj:ds.table")
    .withFormatFunction(e -> new TableRow().set("id", e.getId()).set("ts", e.getTs())));

Write<MyEvent> w = BigQueryIO.<MyEvent>write().to(tableSpec);
if (w.getFormatFunction() == null && w.getAvroFormatFunction() == null && !hasBeamSchema(MyEvent.class)) {
  throw new IllegalStateException("Provide withFormatFunction or a Beam schema before expanding");
}

static boolean needsFormatFunction(Class<?> inputType) {
  return !TableRow.class.isAssignableFrom(inputType) && !GenericRecord.class.isAssignableFrom(inputType);
}

try {
  rows.apply("WriteBQ", writeTransform);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("A function must be provided")) {
    throw new IllegalStateException("Configure withFormatFunction/withAvroFormatFunction or a Beam schema", e);
  } else { throw e; }
}

When it happens

Trigger: Calling BigQueryIO.write() (or writeTables/fileLoads variants) with an input PCollection of a custom POJO or arbitrary type without calling withFormatFunction()/withAvroFormatFunction(), without a schema-derived encoding (setSchema/fromBean methods absent), and where dynamicDestinations/avroRowWriterFactory paths were not taken.

Common situations: Pipelines writing custom Java classes to BigQuery where the developer assumed Beam would auto-convert; refactors that removed a format function while keeping a typed input; using the generic write() instead of schema-aware APIs.

Understand the failure class

Background: "missing required argument" and "the following required arguments were not provided": what required-argument errors mean and how to fix them — this error's family across 20 libraries.

Related errors

• .exceptionsVia() is required
• Unsupported format for BigQuery table path: '{linkedResource
• Starting version or timestamp must be specified.
• Reserved field name <field.name()> in user schema.
• Unsupported type <elementType.getType()>

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/ba12378060c642be. Report an issue: GitHub.