apache/beam · error · IllegalArgumentException

Could not infer Beam schema from Avro schema: {avroSchema}

Error message

Could not infer Beam schema from Avro schema: {avroSchema}

What it means

readAvroGenericRecords converts the Avro schema into a Beam Schema via AvroUtils.getSchema. If Beam cannot infer a schema for the given Avro schema (unsupported types or an unusable schema), it returns null and this IllegalArgumentException is thrown.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIO.java:727

   * but with the with addition of making the message attributes available to the ParseFn.
   */
  public static <T> Read<T> readMessagesWithAttributesWithCoderAndParseFn(
      Coder<T> coder, SimpleFunction<PubsubMessage, T> parseFn) {
    return Read.newBuilder(parseFn).setCoder(coder).setNeedsAttributes(true).build();
  }

  /**
   * Returns a {@link PTransform} that continuously reads binary encoded Avro messages into the Avro
   * {@link GenericRecord} type.
   *
   * <p>Beam will infer a schema for the Avro schema. This allows the output to be used by SQL and
   * by the schema-transform library.
   */
  public static Read<GenericRecord> readAvroGenericRecords(org.apache.avro.Schema avroSchema) {
    AvroCoder<GenericRecord> coder = AvroCoder.of(avroSchema);
    Schema schema = AvroUtils.getSchema(GenericRecord.class, avroSchema);
    if (schema == null) {
      throw new IllegalArgumentException(
          "Could not infer Beam schema from Avro schema: " + avroSchema);
    }
    return Read.newBuilder(parsePayloadUsingCoder(coder))
        .setCoder(
            SchemaCoder.of(
                schema,
                TypeDescriptor.of(GenericRecord.class),
                AvroUtils.getToRowFunction(GenericRecord.class, avroSchema),
                AvroUtils.getFromRowFunction(GenericRecord.class)))
        .build();
  }

  /**
   * Returns a {@link PTransform} that continuously reads binary encoded Avro messages of the
   * specific type.
   *
   * <p>Beam will infer a schema for the Avro schema. This allows the output to be used by SQL and
   * by the schema-transform library.

View on GitHub (pinned to 12126d8942)

Solutions

1. Simplify or fix the Avro schema so all fields map to Beam types (avoid unsupported logical types/unions).
2. Instead of readAvroGenericRecords, use readAvros(specificClass) with a generated Avro class so Beam infers the schema from the class.
3. Parse the payload manually with AvroCoder and MapElements into your own typed PCollection if schema inference is not required.

Example fix

// before
Schema avroSchema = new Schema.Parser().parse(exoticSchemaJson);
PubsubIO.readAvroGenericRecords(avroSchema);
// after
PubsubIO.readAvros(com.example.generated.Event.class); // Beam infers schema from class

Schema beamSchema = AvroUtils.getSchema(GenericRecord.class, avroSchema);
if (beamSchema == null) {
  throw new IllegalArgumentException("Beam cannot map Avro schema; simplify before readAvroGenericRecords");
}
PubsubIO.readAvroGenericRecords(avroSchema);

boolean beamCanMap(org.apache.avro.Schema avroSchema) {
  return AvroUtils.getSchema(GenericRecord.class, avroSchema) != null;
}

try {
  return PubsubIO.readAvroGenericRecords(avroSchema);
} catch (IllegalArgumentException e) {
  if (e.getMessage().startsWith("Could not infer Beam schema from Avro schema"))
    throw new ConfigException("Unsupported Avro schema for Beam mapping", e);
  throw e;
}

When it happens

Trigger: Calling PubsubIO.readAvroGenericRecords(avroSchema) with an Avro schema containing constructs AvroUtils cannot map to a Beam Schema (e.g. unsupported logical types or exotic unions).

Common situations: Avro schemas generated by third-party tools with unusual logical types; deeply nested unions; passing a primitive (non-record) schema that has no mappable Beam representation.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Local timestamp (micros) can only be used with an underlying
• Unknown logical type " + identifier
• Could not infer Beam schema for class: {clazz}
• Unable to generate coder for schema {schema}
• Expecting exactly one field, found

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/7818264a05ee8c53. Report an issue: GitHub.