apache/beam · error · IllegalArgumentException

configuration with compression is not compatible with AvroIO

Error message

configuration with compression is not compatible with AvroIO

What it means

validateConfiguration rejects a compression setting when the format is AVRO, because the AvroIO sink in this transform does not accept a user-supplied compression. It is an IllegalArgumentException raised at validation time before the pipeline runs.

Solutions

1. Remove/unset compression when format is AVRO
2. Set compression only for PARQUET format
3. Use AvroIO-specific write settings outside this transform if compression is required

Example fix

// before
builder.setFormat("AVRO").setCompression("SNAPPY");
// after
builder.setFormat("AVRO").setCompression(null); // or setFormat("PARQUET") to keep SNAPPY

if ("AVRO".equals(cfg.getFormat()) && cfg.getCompression() != null && !cfg.getCompression().isEmpty()) throw new IllegalArgumentException("compression not allowed for AVRO");

try { transform.expand(p.apply(...)); } catch (IllegalArgumentException e) { if (e.getMessage().contains("compression")) { log.warn("strip compression for AVRO"); } else throw e; }

When it happens

Trigger: FileWriteSchemaTransformConfiguration with format=AVRO and a non-empty compression (e.g. setCompression("SNAPPY") or "GZIP").

Common situations: Reusing the same configuration object across formats where compression was set for Parquet; assuming Avro supports the same compression knob as Parquet in this transform.

Related errors

• configuration with compression is not compatible with…
• Batch size is too large! It should be smaller or equal than
• Both clientCertPath and clientCertKeyPath must be specified…
• ChangeStreamName can't be empty
• Column delimiter should be set if headers are present.

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/30c8ceedd9bf7b1e. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/file-schema-transform/src/main/java/org/apache/beam/sdk/io/fileschematransform/FileWriteSchemaTransformProvider.java:193

            String.format(
                "configuration with %s is not compatible with a %s format",
                FileWriteSchemaTransformConfiguration.CsvConfiguration.class.getName(), format));
      }
      if (configuration.getParquetConfiguration() != null && !format.equals(PARQUET)) {
        throw new IllegalArgumentException(
            String.format(
                "configuration with %s is not compatible with a %s format",
                FileWriteSchemaTransformConfiguration.ParquetConfiguration.class.getName(),
                format));
      }
      if (configuration.getXmlConfiguration() != null && !format.equals(XML)) {
        throw new IllegalArgumentException(
            String.format(
                "configuration with %s is not compatible with a %s format",
                FileWriteSchemaTransformConfiguration.XmlConfiguration.class.getName(), format));
      }
      if (format.equals(AVRO) && !Strings.isNullOrEmpty(configuration.getCompression())) {
        throw new IllegalArgumentException(
            "configuration with compression is not compatible with AvroIO");
      }
      if (format.equals(PARQUET) && !Strings.isNullOrEmpty(configuration.getCompression())) {
        throw new IllegalArgumentException(
            "configuration with compression is not compatible with ParquetIO");
      }
    }
  }
}

View on GitHub (pinned to 12126d8942)