What causes "file modifier options not compatible with stream" in apache/hadoop?

Remove the file modifier option(s) when using Writer.stream

apache/hadoop · error · IllegalArgumentException

file modifier options not compatible with stream

Error message

file modifier options not compatible with stream

What it means

When SequenceFile.createWriter writes to a caller-supplied stream (Writer.stream(...)), filesystem-level knobs are meaningless because the writer never opens a file: any of Writer.blockSize, Writer.bufferSize, Writer.replication, or Writer.progressable combined with a stream option throws this IllegalArgumentException at construction.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:1104

      KeyClassOption keyClassOption = 
        Options.getOption(KeyClassOption.class, opts);
      ValueClassOption valueClassOption = 
        Options.getOption(ValueClassOption.class, opts);
      MetadataOption metadataOption = 
        Options.getOption(MetadataOption.class, opts);
      CompressionOption compressionTypeOption =
        Options.getOption(CompressionOption.class, opts);
      SyncIntervalOption syncIntervalOption =
          Options.getOption(SyncIntervalOption.class, opts);
      // check consistency of options
      if ((fileOption == null) == (streamOption == null)) {
        throw new IllegalArgumentException("file or stream must be specified");
      }
      if (fileOption == null && (blockSizeOption != null ||
                                 bufferSizeOption != null ||
                                 replicationOption != null ||
                                 progressOption != null)) {
        throw new IllegalArgumentException("file modifier options not " +
                                           "compatible with stream");
      }

      FSDataOutputStream out;
      boolean ownStream = fileOption != null;
      if (ownStream) {
        Path p = fileOption.getValue();
        FileSystem fs;
        if (fsOption != null) {
          fs = fsOption.getValue();
        } else {
          fs = p.getFileSystem(conf);
        }
        int bufferSize = bufferSizeOption == null ? getBufferSize(conf) :
          bufferSizeOption.getValue();
        short replication = replicationOption == null ? 
          fs.getDefaultReplication(p) :
          (short) replicationOption.getValue();

View on GitHub (pinned to 2add963021)

Solutions

Remove the file modifier option(s) when using Writer.stream
If blockSize/replication/bufferSize tuning is required, switch to Writer.file(path) and let the writer open the file
Apply buffering to the stream itself before passing it (e.g. wrap in a BufferedOutputStream) instead of using Writer.bufferSize

Example fix

// before
Writer w = SequenceFile.createWriter(conf, Writer.stream(out),
    Writer.bufferSize(64 * 1024), Writer.replication((short) 3), // throws
    Writer.keyClass(Text.class), Writer.valueClass(Text.class));

// after
Writer w = SequenceFile.createWriter(conf, Writer.stream(out),
    Writer.keyClass(Text.class), Writer.valueClass(Text.class));

Defensive patterns

Strategy: validation

Validate before calling

if (usingStream) {
  // only type/metadata/key/value/syncInterval options are legal; assert no file modifiers
  for (SequenceFile.Writer.Option o : opts) {
    if (o instanceof SequenceFile.Writer.blockSizeOption) throw new IllegalArgumentException("blockSize with stream");
  }
}

Prevention

Keep two separate option builders: fileTarget(path, blockSize...) and streamTarget(out)
Apply stream-level buffering yourself instead of Writer.bufferSize when writing to a stream

When it happens

Trigger: createWriter(conf, Writer.stream(out), Writer.blockSize(128*1024*1024), ...) or any other single file-modifier option alongside Writer.stream.

Common situations: Refactoring legacy code that created its own FSDataOutputStream while keeping tuning options from the file-based version; generic writer factories that unconditionally attach a progressable; test harnesses wrapping in-memory streams.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/c1889d8ab0c23e66. Report an issue: GitHub.