apache/hadoop · error · IOException

closed has error. bs : pre write obs[%s] has error.

Error message

closed has error. bs : pre write obs[%s] has error.

What it means

OBSBlockOutputStream.close() flips the closed flag, then checks hasException: if any prior operation on the stream failed, close() throws IOException('closed has error. bs : pre write obs[<key>] has error.') instead of performing the final multipart complete. The object is therefore NOT finalized in OBS — data may be partially uploaded — and the caller must treat the file as failed. As with write(), the informative exception happened earlier; this one only signals 'do not trust this file'.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java:417

   *
   * <p>This will not return until the upload is complete or the attempt to
   * perform the upload has failed. Exceptions raised in this method are
   * indicative that the write has failed and data is at risk of being lost.
   *
   * @throws IOException on any failure.
   */
  @Override
  public synchronized void close() throws IOException {
    if (closed.getAndSet(true)) {
      // already closed
      LOG.debug("Ignoring close() as stream is already closed");
      return;
    }
    if (hasException.get()) {
      String closeWarning = String.format(
          "closed has error. bs : pre write obs[%s] has error.", key);
      LOG.warn(closeWarning);
      throw new IOException(closeWarning);
    }
    // do upload
    completeCurrentBlock();

    // clear
    clearHFlushOrSync();

    // All end of write operations, including deleting fake parent
    // directories
    writeOperationHelper.writeSuccessful(key);
  }

  /**
   * If flush has take place, need to append file, else to put object.
   *
   * @throws IOException any problem in append or put object
   */
  private synchronized void putObjectIfNeedAppend() throws IOException {

View on GitHub (pinned to 2add963021)

Solutions

  1. Treat this as a failed write: locate the FIRST exception for the key in logs (network/auth/quota) and fix that.
  2. Ensure the multipart upload is aborted to avoid orphaned parts billing: call ((OBSBlockOutputStream) out.getWrappedStream()).abort() in the catch, or rely on the filesystem's abort hooks; verify with the OBS console/lifecycle rule for incomplete MPU cleanup.
  3. Retry the entire file write after the root cause is fixed — there is no way to resume a stream after this error.
  4. Harden close paths: catch IOException from close() separately from the body so both original and close failures are reported.

Example fix

// before
try (FSDataOutputStream out = fs.create(path)) {
  writeAll(out, records); // earlier part-upload failure logged here
} // close() now throws 'closed has error'

// after
FSDataOutputStream out = fs.create(path);
try {
  writeAll(out, records);
  out.close();
} catch (IOException e) {
  try { ((OBSBlockOutputStream) out.getWrappedStream()).abort(); } catch (IOException ignore) {}
  throw new IOException("Write failed for " + path + " - see earlier root-cause exception", e);
}
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check your own failure flag before close
if (writeFailed) {
  abortQuietly(out);
} else {
  out.close(); // may still throw if a background upload failed silently
}

Try / catch

try {
  out.close();
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("pre write obs[")) {
    abortQuietly(out); // ensure multipart is aborted
    LOG.error("object NOT finalized in OBS; treating write as failed", e);
    throw new WriteFailedException(path, e);
  }
  throw e;
}
// helper
void abortQuietly(FSDataOutputStream o) {
  try { if (o.getWrappedStream() instanceof OBSBlockOutputStream) ((OBSBlockOutputStream) o.getWrappedStream()).abort(); } catch (IOException ignore) {}
}

Prevention

When it happens

Trigger: Any earlier write/flush/block-upload failure followed by a normal try-with-resources close(); background upload thread failing while the main thread finishes writing and closes; the completeCurrentBlock() or a prior part upload aborting asynchronously before close is reached.

Common situations: Jobs whose real error was logged minutes before the misleading 'close failed' symptom at task commit; monitoring that only surfaces the close() exception and misses the first stack; users assuming close() will 'flush through' problems and salvage partial data.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/699966fedb6179a4. Report an issue: GitHub.