apache/flink · error · IllegalStateException

Schema must be set when using Generic Record

Error message

Schema must be set when using Generic Record

What it means

Thrown by AvroOutputFormat when the configured record class implements org.apache.avro.generic.GenericRecord but no Avro schema was supplied. Generic records carry no embedded schema (unlike SpecificRecord, whose schema is obtained by instantiating the class), so the output format has nothing to write against and fails fast during open().

Source

Thrown at flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroOutputFormat.java:140

    @Override
    public void open(InitializationContext context) throws IOException {
        super.open(context);

        DatumWriter<E> datumWriter;
        Schema schema;
        if (org.apache.avro.specific.SpecificRecordBase.class.isAssignableFrom(avroValueType)) {
            datumWriter = new SpecificDatumWriter<E>(avroValueType);
            try {
                schema =
                        ((org.apache.avro.specific.SpecificRecordBase) avroValueType.newInstance())
                                .getSchema();
            } catch (InstantiationException | IllegalAccessException e) {
                throw new RuntimeException(e.getMessage());
            }
        } else if (org.apache.avro.generic.GenericRecord.class.isAssignableFrom(avroValueType)) {
            if (userDefinedSchema == null) {
                throw new IllegalStateException("Schema must be set when using Generic Record");
            }
            datumWriter = new GenericDatumWriter<E>(userDefinedSchema);
            schema = userDefinedSchema;
        } else {
            datumWriter = new ReflectDatumWriter<E>(avroValueType);
            schema = ReflectData.get().getSchema(avroValueType);
        }
        dataFileWriter = new DataFileWriter<E>(datumWriter);
        if (codec != null) {
            dataFileWriter.setCodec(codec.getCodecFactory());
        }
        if (userDefinedSchema == null) {
            dataFileWriter.create(schema, stream);
        } else {
            dataFileWriter.create(userDefinedSchema, stream);
        }
    }

View on GitHub (pinned to 2f3c205e92)

Solutions

1. Pass an explicit org.apache.avro.Schema to the AvroOutputFormat constructor (or use the builder/factory overload that accepts a schema string) so userDefinedSchema is non-null.
2. If you have a generated class, use the Avro-codegen'd SpecificRecord subclass instead of a raw GenericRecord so the schema is auto-derived.
3. If the schema lives in a file/classpath resource, load and parse it with new Schema.Parser().parse(...) and hand it to the format before open().

Example fix

// before
AvroOutputFormat<GenericRecord> out =
    new AvroOutputFormat<>(new Path(path), GenericRecord.class);

// after
Schema schema = new Schema.Parser().parse(
    new InputStreamReader(getClass().getResourceAsStream("/event.avsc")));
AvroOutputFormat<GenericRecord> out =
    new AvroOutputFormat<>(new Path(path), GenericRecord.class, schema);

if (GenericRecord.class.isAssignableFrom(recordClass)
        && !SpecificRecordBase.class.isAssignableFrom(recordClass)
        && schema == null) {
    throw new IllegalArgumentException("AvroOutputFormat needs an explicit Schema for GenericRecord types: "
        + recordClass.getName());
}

static boolean needsExplicitSchema(Class<?> c, Schema s) {
    return GenericRecord.class.isAssignableFrom(c)
            && !SpecificRecordBase.class.isAssignableFrom(c)
            && s == null;
}

When it happens

Trigger: new AvroOutputFormat<>(GenericRecord.class (e.g. IndexedRecord/DataFileWriter path), codec) or AvroOutputFormat.writeAvro(...)'s overload that takes no schema, where the value type is GenericRecord but not SpecificRecordBase; the constructor's userDefinedSchema stays null.

Common situations: Switching a job from an Avro-generated SpecificRecord class to GenericData.Record or AvroUtils-supported generic records without adding a schema; using the deprecated no-schema factory method; refactoring tuple->pojo->generic avro types.

Related errors

• Unsupported type: {}
• Avro Union with NULL type is only supported. Unsupported typ
• The Avro schema is not a nullable type: %s
• Could not parse Avro schema string.
• Unsupported Avro type '%s'.

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/6542c7286a36d7ed. Report an issue: GitHub.