apache/hadoop · error · IllegalArgumentException

file or stream must be specified

Error message

file or stream must be specified

What it means

SequenceFile.createWriter(conf, Option...) requires exactly one destination option: either Writer.file(path) or Writer.stream(FSDataOutputStream). The constructor checks (fileOption == null) == (streamOption == null) and throws this IllegalArgumentException when both are present or both are absent, failing fast before any output is opened.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java:1098

        Options.getOption(ProgressableOption.class, opts);
      FileOption fileOption = Options.getOption(FileOption.class, opts);
      AppendIfExistsOption appendIfExistsOption = Options.getOption(
          AppendIfExistsOption.class, opts);
      FileSystemOption fsOption = Options.getOption(FileSystemOption.class, opts);
      StreamOption streamOption = Options.getOption(StreamOption.class, opts);
      KeyClassOption keyClassOption = 
        Options.getOption(KeyClassOption.class, opts);
      ValueClassOption valueClassOption = 
        Options.getOption(ValueClassOption.class, opts);
      MetadataOption metadataOption = 
        Options.getOption(MetadataOption.class, opts);
      CompressionOption compressionTypeOption =
        Options.getOption(CompressionOption.class, opts);
      SyncIntervalOption syncIntervalOption =
          Options.getOption(SyncIntervalOption.class, opts);
      // check consistency of options
      if ((fileOption == null) == (streamOption == null)) {
        throw new IllegalArgumentException("file or stream must be specified");
      }
      if (fileOption == null && (blockSizeOption != null ||
                                 bufferSizeOption != null ||
                                 replicationOption != null ||
                                 progressOption != null)) {
        throw new IllegalArgumentException("file modifier options not " +
                                           "compatible with stream");
      }

      FSDataOutputStream out;
      boolean ownStream = fileOption != null;
      if (ownStream) {
        Path p = fileOption.getValue();
        FileSystem fs;
        if (fsOption != null) {
          fs = fsOption.getValue();
        } else {
          fs = p.getFileSystem(conf);

View on GitHub (pinned to 2add963021)

Solutions

  1. Pass exactly one of Writer.file(path) or Writer.stream(out)
  2. If the target is a path, remove the stream option (and note the writer will open/close the file itself)
  3. If the target is a stream, remove the file option and also drop file modifier options like blockSize/bufferSize/replication/progressable

Example fix

// before
Writer w = SequenceFile.createWriter(conf,
    Writer.file(p), Writer.stream(out),
    Writer.keyClass(Text.class), Writer.valueClass(Text.class)); // throws

// after
Writer w = SequenceFile.createWriter(conf,
    Writer.file(p),
    Writer.keyClass(Text.class), Writer.valueClass(Text.class));
Defensive patterns

Strategy: validation

Validate before calling

// assemble options so exactly one destination is ever added
SequenceFile.Writer.Option[] opts;
if (out != null) {
  opts = new Option[]{ Writer.stream(out), Writer.keyClass(kc), Writer.valueClass(vc) };
} else {
  opts = new Option[]{ Writer.file(path), Writer.keyClass(kc), Writer.valueClass(vc) };
}

Prevention

When it happens

Trigger: Passing both Writer.file(p) and Writer.stream(out) in the same option list; passing only key/value/compression options with no destination at all.

Common situations: Migrating legacy SequenceFile.createWriter(Path, ...) call sites onto the Options API; copy-pasted option lists where a stream option survived next to a new file option; factory code that assembles options dynamically and ends up with none.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1199462119bffc9a. Report an issue: GitHub.