apache/hadoop · error · IOException

write has error. bs : pre upload obs[%s] has error.

Error message

write has error. bs : pre upload obs[%s] has error.

What it means

OBSBlockOutputStream.write() checks the hasException AtomicBoolean: once any earlier block upload or internal operation failed, every subsequent write() fails fast with IOException('write has error. bs : pre upload obs[<key>] has error.'). This is a secondary, deliberately uninformative error — the real failure was logged when hasException was set. Its purpose is to stop the write path from piling more data onto a stream that is already known-broken.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java:307

  /**
   * Writes a range of bytes from to the memory buffer. If this causes the
   * buffer to reach its limit, the actual upload is submitted to the threadpool
   * and the remainder of the array is written to memory (recursively).
   *
   * @param source byte array containing
   * @param offset offset in array where to start
   * @param len    number of bytes to be written
   * @throws IOException on any problem
   */
  @Override
  public synchronized void write(@NotNull final byte[] source,
      final int offset, final int len)
      throws IOException {
    if (hasException.get()) {
      String closeWarning = String.format(
          "write has error. bs : pre upload obs[%s] has error.", key);
      LOG.warn(closeWarning);
      throw new IOException(closeWarning);
    }
    OBSDataBlocks.validateWriteArgs(source, offset, len);
    checkOpen();
    if (len == 0) {
      return;
    }

    OBSDataBlocks.DataBlock block = createBlockIfNeeded();
    int written = block.write(source, offset, len);
    int remainingCapacity = block.remainingCapacity();
    try {
      innerWrite(source, offset, len, written, remainingCapacity);
    } catch (IOException e) {
      LOG.error(
          "Write data for key {} of bucket {} error, error message {}",
          key, fs.getBucket(),
          e.getMessage());
      throw e;

View on GitHub (pinned to 2add963021)

Solutions

  1. Scroll UP in the logs to the first exception for this key — that root cause (auth, network, quota) is what to fix; this exception is only the messenger.
  2. Close/abort the stream and fail the task; then retry the whole write from the last committed point (multipart parts already uploaded are discarded on abort).
  3. If root cause is transient (5xx, connection reset), enable/verify fs.obs.attempts and retry policy on the client and retry the job.
  4. If root cause is credentials, refresh AK/SK or session token before the retry; expired temporary credentials are the most common cause in long jobs.

Example fix

// before
try { out.write(buf, 0, n); } catch (IOException e) { log.warn("write failed, keep going"); }
out.write(more); // -> 'pre upload obs[...] has error'

// after
try {
  out.write(buf, 0, n);
} catch (IOException e) {
  quietlyAbort(out); // ((OBSBlockOutputStream) out.getWrappedStream()).abort() or close quietly
  throw new RuntimeException("OBS write failed for " + path, e); // fail task, retry job
}
Defensive patterns

Strategy: try-catch

Validate before calling

// no public isFailed() exists; track the first failure in your wrapper
class SafeObsWriter {
  private volatile boolean failed;
  void write(FSDataOutputStream out, byte[] b, int off, int len) throws IOException {
    if (failed) throw new IllegalStateException("stream already failed");
    try { out.write(b, off, len); }
    catch (IOException e) { failed = true; throw e; }
  }
}

Try / catch

try {
  out.write(buf, 0, n);
} catch (IOException e) {
  if (String.valueOf(e.getMessage()).contains("pre upload obs[")) {
    abortQuietly(out);
    throw new RuntimeException("OBS stream poisoned by earlier failure — see first exception in logs", e);
  }
  throw e;
}

Prevention

When it happens

Trigger: A background block upload (multipart part) failed earlier — network reset, 401/403 expired credentials, 5xx from OBS, out-of-memory in the buffering block factory — and the caller ignores the first exception and keeps writing; async failure happens between two write() calls, so the next write() sees hasException==true; previous flush() swallowed an exception in an outer wrapper.

Common situations: Long-running Spark/Hive writes where a transient network blip failed one part upload; AK/SK revocation mid-job; executors with tight memory causing block allocation failures; callers that catch-and-continue around write loops.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/d3dbc2954aaf8690. Report an issue: GitHub.