{"record":{"id":"c1889d8ab0c23e66","repo":"apache/hadoop","slug":"file-modifier-options-not-compatible-with-stream","errorCode":null,"errorMessage":"file modifier options not compatible with stream","messagePattern":"file modifier options not compatible with stream","errorType":"exception","errorClass":"IllegalArgumentException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java","lineNumber":1104,"sourceCode":"      KeyClassOption keyClassOption = \n        Options.getOption(KeyClassOption.class, opts);\n      ValueClassOption valueClassOption = \n        Options.getOption(ValueClassOption.class, opts);\n      MetadataOption metadataOption = \n        Options.getOption(MetadataOption.class, opts);\n      CompressionOption compressionTypeOption =\n        Options.getOption(CompressionOption.class, opts);\n      SyncIntervalOption syncIntervalOption =\n          Options.getOption(SyncIntervalOption.class, opts);\n      // check consistency of options\n      if ((fileOption == null) == (streamOption == null)) {\n        throw new IllegalArgumentException(\"file or stream must be specified\");\n      }\n      if (fileOption == null && (blockSizeOption != null ||\n                                 bufferSizeOption != null ||\n                                 replicationOption != null ||\n                                 progressOption != null)) {\n        throw new IllegalArgumentException(\"file modifier options not \" +\n                                           \"compatible with stream\");\n      }\n\n      FSDataOutputStream out;\n      boolean ownStream = fileOption != null;\n      if (ownStream) {\n        Path p = fileOption.getValue();\n        FileSystem fs;\n        if (fsOption != null) {\n          fs = fsOption.getValue();\n        } else {\n          fs = p.getFileSystem(conf);\n        }\n        int bufferSize = bufferSizeOption == null ? getBufferSize(conf) :\n          bufferSizeOption.getValue();\n        short replication = replicationOption == null ? \n          fs.getDefaultReplication(p) :\n          (short) replicationOption.getValue();","sourceCodeStart":1086,"sourceCodeEnd":1122,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/SequenceFile.java#L1086-L1122","documentation":"When SequenceFile.createWriter writes to a caller-supplied stream (Writer.stream(...)), filesystem-level knobs are meaningless because the writer never opens a file: any of Writer.blockSize, Writer.bufferSize, Writer.replication, or Writer.progressable combined with a stream option throws this IllegalArgumentException at construction.","triggerScenarios":"createWriter(conf, Writer.stream(out), Writer.blockSize(128*1024*1024), ...) or any other single file-modifier option alongside Writer.stream.","commonSituations":"Refactoring legacy code that created its own FSDataOutputStream while keeping tuning options from the file-based version; generic writer factories that unconditionally attach a progressable; test harnesses wrapping in-memory streams.","solutions":["Remove the file modifier option(s) when using Writer.stream","If blockSize/replication/bufferSize tuning is required, switch to Writer.file(path) and let the writer open the file","Apply buffering to the stream itself before passing it (e.g. wrap in a BufferedOutputStream) instead of using Writer.bufferSize"],"exampleFix":"// before\nWriter w = SequenceFile.createWriter(conf, Writer.stream(out),\n    Writer.bufferSize(64 * 1024), Writer.replication((short) 3), // throws\n    Writer.keyClass(Text.class), Writer.valueClass(Text.class));\n\n// after\nWriter w = SequenceFile.createWriter(conf, Writer.stream(out),\n    Writer.keyClass(Text.class), Writer.valueClass(Text.class));","handlingStrategy":"validation","validationCode":"if (usingStream) {\n  // only type/metadata/key/value/syncInterval options are legal; assert no file modifiers\n  for (SequenceFile.Writer.Option o : opts) {\n    if (o instanceof SequenceFile.Writer.blockSizeOption) throw new IllegalArgumentException(\"blockSize with stream\");\n  }\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep two separate option builders: fileTarget(path, blockSize...) and streamTarget(out)","Apply stream-level buffering yourself instead of Writer.bufferSize when writing to a stream"],"tags":["hadoop","sequence-file","api-misuse","invalid-argument"],"backgroundTag":"invalid-argument-combination","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}