apache/beam · error · IOException

Couldn't set the specified Avro data model

Error message

Couldn't set the specified Avro data model %s

What it means

ParquetIO's sink builder allows specifying an Avro data model class (e.g. SpecificRecord or ReflectData models). When instantiating the model object reflectively fails (class missing, no suitable constructor, access error), it rethrows as an IOException with the model class name.

Solutions

1. Ensure the model class (and its Avro-generated dependencies) is on the worker classpath.
2. Verify the class has an accessible constructor expected by AvroData.newModel (e.g. no-arg for SpecificData).
3. Check for shading/relocation of org.apache.avro that separates the model class from the avro runtime.
4. Fall back to the default model (GenericData) if reflection-based models are unnecessary.

Example fix

// before
ParquetIO.sink(schema).withDataModel(SpecificData.class) // class shaded away on worker
// after
ParquetIO.sink(schema) // default GenericData model, or ensure SpecificData is bundled un-relocated

// Verify model class is instantiable before configuring the sink
Class<?> modelClass = SpecificData.class;
try { modelClass.getDeclaredConstructor().newInstance(); } catch (ReflectiveOperationException e) { throw new IllegalStateException("model not usable: " + modelClass, e); }

try { pipeline.apply("write", ParquetIO.sink(schema).withDatumWriter(...)); } catch (IOException e) { if (e.getMessage().startsWith("Couldn't set the specified Avro data model")) { /* fall back to default model */ } }

When it happens

Trigger: Calling ParquetIO.sink(...).withDatumWriter(...)/withDataModel equivalent by configuring a modelClass whose instance cannot be built via buildModelObject (ReflectiveOperationException).

Common situations: Specifying a SpecificRecord class not on the runtime classpath (fat-jar shading issues); model class lacking a no-arg constructor; using GenericData where SpecificData/ReflectData is needed.

Related errors

• A schema is required to write non-schema'd data.
• A schema must be provided when writing to BigQuery using…
• An explicit schema is required to write non-schema'd…
• AutoValue builder class
• AvroCoder for GenericRecord requires a schema

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/b79b55516d343be3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java:1224

          checkArgument(minRowCount > 0, "minRowCountForPageSizeCheck must be positive");
          builder = builder.withMinRowCountForPageSizeCheck(minRowCount);
        }
      }

      ValueProvider<Integer> maxRowCountProvider = getMaxRowCountForPageSizeCheck();
      if (maxRowCountProvider != null) {
        Integer maxRowCount = maxRowCountProvider.get();
        if (maxRowCount != null) {
          checkArgument(maxRowCount > 0, "maxRowCountForPageSizeCheck must be positive");
          builder = builder.withMaxRowCountForPageSizeCheck(maxRowCount);
        }
      }

      if (modelClass != null) {
        try {
          builder.withDataModel(buildModelObject(modelClass));
        } catch (ReflectiveOperationException e) {
          throw new IOException(
              "Couldn't set the specified Avro data model " + modelClass.getName(), e);
        }
      }
      this.writer = builder.build();
    }

    @Override
    public void write(GenericRecord element) throws IOException {
      checkNotNull(writer, "Writer cannot be null");
      writer.write(element);
    }

    @Override
    public void flush() throws IOException {
      // the only way to completely flush the output is to call writer.close() here
      writer.close();
    }

View on GitHub (pinned to 12126d8942)