What causes "%s closed" in apache/hadoop?

Create a new encoder (CodecUtil.createRawEncoder / RawErasureCoderFactory) instead of reusing a closed one

apache/hadoop · error · IOException

%s closed

Error message

%s closed

What it means

Native raw encoders (ISA-L backed, e.g. NativeRSRawEncoder) keep a long nativeCoder handle that becomes 0 once close() releases the native coder. A later doEncode sees nativeCoder == 0 and throws IOException("<ClassName> closed"): the encoder instance is a closed native resource and must not be reused.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/rawcoder/AbstractNativeRawEncoder.java:53

  public static Logger LOG =
      LoggerFactory.getLogger(AbstractNativeRawEncoder.class);

  // Protect ISA-L coder data structure in native layer from being accessed and
  // updated concurrently by the init, release and encode functions.
  protected final ReentrantReadWriteLock encoderLock =
      new ReentrantReadWriteLock();

  public AbstractNativeRawEncoder(ErasureCoderOptions coderOptions) {
    super(coderOptions);
  }

  @Override
  protected void doEncode(ByteBufferEncodingState encodingState)
      throws IOException {
    encoderLock.readLock().lock();
    try {
      if (nativeCoder == 0) {
        throw new IOException(String.format("%s closed",
            getClass().getSimpleName()));
      }
      int[] inputOffsets = new int[encodingState.inputs.length];
      int[] outputOffsets = new int[encodingState.outputs.length];
      int dataLen = encodingState.inputs[0].remaining();

      ByteBuffer buffer;
      for (int i = 0; i < encodingState.inputs.length; ++i) {
        buffer = encodingState.inputs[i];
        inputOffsets[i] = buffer.position();
      }

      for (int i = 0; i < encodingState.outputs.length; ++i) {
        buffer = encodingState.outputs[i];
        outputOffsets[i] = buffer.position();
      }

      performEncodeImpl(encodingState.inputs, inputOffsets, dataLen,

View on GitHub (pinned to 2add963021)

Solutions

Create a new encoder (CodecUtil.createRawEncoder / RawErasureCoderFactory) instead of reusing a closed one
Match encoder lifetime to workload scope; never let pools return closed instances
On any exception path that closes an encoder, evict it from caches immediately

Example fix

// before
try (RawErasureEncoder enc = CodecUtil.createRawEncoder(conf, "rs", opts)) {
  enc.encode(inputs, outputs);
}
enc.encode(moreInputs, moreOutputs); // reused after close -> IOException

// after
RawErasureEncoder enc = CodecUtil.createRawEncoder(conf, "rs", opts);
enc.encode(inputs, outputs);
enc.encode(moreInputs, moreOutputs);
enc.close(); // close only when truly done

Defensive patterns

Strategy: try-catch

Validate before calling

// Keep encoder lifetime explicit: one encoder per workload batch
RawErasureEncoder encoder = CodecUtil.createRawEncoder(conf, "rs", opts);
try {
  encoder.encode(inputs, outputs);
} finally {
  encoder.close(); // close only when no more encodes are needed
}

Try / catch

try {
  encoder.encode(inputs, outputs);
} catch (IOException e) {
  if (e.getMessage().endsWith("closed")) {
    encoder = CodecUtil.createRawEncoder(conf, "rs", opts);
    encoder.encode(inputs, outputs); // one retry with a fresh encoder
  } else throw e;
}

Prevention

Do not wrap pooled/long-lived native encoders in try-with-resources
On exception paths that close an encoder, always remove it from caches
Log close() calls on shared coders during development to catch lifecycle bugs

When it happens

Trigger: Calling encode() on a native RawErasureEncoder that was already closed — via explicit close(), try-with-resources on a shared/pooled instance, or a cleanup hook racing with subsequent encodes.

Common situations: Encoder pools whose error handling closes instances still in circulation; wrapping long-lived encoders in try-with-resources; retry logic that reuses an encoder after an earlier failure path closed it.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/879d61497734ebc6. Report an issue: GitHub.