{"record":{"id":"173c78b571e81227","repo":"apache/hadoop","slug":"compress-called-on-finished-compressor","errorCode":null,"errorMessage":"compress called on finished compressor","messagePattern":"compress called on finished compressor","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java","lineNumber":84,"sourceCode":"  public BuiltInGzipCompressor(Configuration conf) {\n    init(conf);\n  }\n\n  @Override\n  public boolean finished() {\n    // Only if the trailer is also written, it is thought as finished.\n    return state == BuiltInGzipDecompressor.GzipStateLabel.FINISHED && deflater.finished();\n  }\n\n  @Override\n  public boolean needsInput() {\n    return deflater.needsInput() && state != BuiltInGzipDecompressor.GzipStateLabel.TRAILER_CRC;\n  }\n\n  @Override\n  public int compress(byte[] b, int off, int len) throws IOException {\n    if (finished()) {\n      throw new IOException(\"compress called on finished compressor\");\n    }\n\n    if (state == BuiltInGzipDecompressor.GzipStateLabel.ENDED) {\n      throw new AlreadyClosedException(\"compress called on closed compressor\");\n    }\n\n    int compressedBytesWritten = 0;\n\n    // If we are not within uncompressed data yet, output the header.\n    if (state == BuiltInGzipDecompressor.GzipStateLabel.HEADER_BASIC) {\n      int outputHeaderSize = writeHeader(b, off, len);\n      numExtraBytesWritten += outputHeaderSize;\n\n      compressedBytesWritten += outputHeaderSize;\n\n      if (outputHeaderSize == len) {\n        return compressedBytesWritten;\n      }","sourceCodeStart":66,"sourceCodeEnd":102,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/BuiltInGzipCompressor.java#L66-L102","documentation":"BuiltInGzipCompressor is Hadoop's pure-Java gzip Compressor (used when zlib natives are unavailable and io.compression.codec.gzip.no.native handles the choice). finished() returns true only after the gzip trailer (CRC/size) has been emitted; compress(byte[], off, len) throws IOException(\"compress called on finished compressor\") if invoked after that point, because a gzip member is complete and cannot accept more input. The compressor must be reset() (or reinit(Configuration)) before compressing another stream/member.","triggerScenarios":"Calling compress(...) on a BuiltInGzipCompressor whose state is FINISHED and deflater.finished(): reusing a pooled codec object for a second file without reset(); CompressorStream-like loops that keep calling compress after the trailer was written; multi-file writers sharing one compressor instance.","commonSituations":"Compressor pools (CodecPool) returning used compressors; frameworks (HDFS sink, Hive writers) that cache codec objects across files/rotations; custom OutputCommitter code writing several gzip outputs through one compressor; tests reusing a static compressor instance.","solutions":["Before each new input, call compressor.reset() (or reinit(conf) when Configuration-derived settings must be re-applied) — pools typically do this via CodecPool.returnCompressor/CompressorPool reset semantics.","Check finished() defensively and reinit when true instead of calling compress on a completed member.","Return compressors to CodecPool after each output and check out a fresh/reinit one per file.","Audit shared/static compressor fields — make the compressor per-output, not global."],"exampleFix":"// before\ncompressor.compress(buf, 0, n); // second file: IOException 'compress called on finished compressor'\n\n// after\nif (compressor.finished()) {\n  compressor.reinit(conf); // or compressor.reset()\n}\ncompressor.compress(buf, 0, n);","handlingStrategy":"validation","validationCode":"if (compressor.finished()) {\n  compressor.reinit(conf); // or compressor.reset() when no config is needed\n}\ncompressor.compress(buf, 0, len);","typeGuard":null,"tryCatchPattern":"try {\n  compressor.compress(buf, 0, len);\n} catch (IOException e) {\n  if (\"compress called on finished compressor\".equals(e.getMessage())) {\n    compressor.reinit(conf);\n    compressor.compress(buf, 0, len); // retry once after reinit\n  } else {\n    throw e;\n  }\n}","preventionTips":["reset()/reinit() every pooled compressor before reuse (CodecPool semantics).","Scope compressor instances to a single output/file rather than sharing statics.","Check finished() in loop drivers between members of multi-file outputs."],"tags":["gzip","zlib","compression","reuse-after-finish","compressor-pool","hadoop"],"backgroundTag":"reuse-after-finish","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}