{"record":{"id":"d3dbc2954aaf8690","repo":"apache/hadoop","slug":"write-has-error-bs-pre-upload-obs-s-has-error","errorCode":null,"errorMessage":"write has error. bs : pre upload obs[%s] has error.","messagePattern":"write has error\\. bs : pre upload obs\\[(.+?)\\] has error\\.","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java","lineNumber":307,"sourceCode":"  /**\n   * Writes a range of bytes from to the memory buffer. If this causes the\n   * buffer to reach its limit, the actual upload is submitted to the threadpool\n   * and the remainder of the array is written to memory (recursively).\n   *\n   * @param source byte array containing\n   * @param offset offset in array where to start\n   * @param len    number of bytes to be written\n   * @throws IOException on any problem\n   */\n  @Override\n  public synchronized void write(@NotNull final byte[] source,\n      final int offset, final int len)\n      throws IOException {\n    if (hasException.get()) {\n      String closeWarning = String.format(\n          \"write has error. bs : pre upload obs[%s] has error.\", key);\n      LOG.warn(closeWarning);\n      throw new IOException(closeWarning);\n    }\n    OBSDataBlocks.validateWriteArgs(source, offset, len);\n    checkOpen();\n    if (len == 0) {\n      return;\n    }\n\n    OBSDataBlocks.DataBlock block = createBlockIfNeeded();\n    int written = block.write(source, offset, len);\n    int remainingCapacity = block.remainingCapacity();\n    try {\n      innerWrite(source, offset, len, written, remainingCapacity);\n    } catch (IOException e) {\n      LOG.error(\n          \"Write data for key {} of bucket {} error, error message {}\",\n          key, fs.getBucket(),\n          e.getMessage());\n      throw e;","sourceCodeStart":289,"sourceCodeEnd":325,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSBlockOutputStream.java#L289-L325","documentation":"OBSBlockOutputStream.write() checks the hasException AtomicBoolean: once any earlier block upload or internal operation failed, every subsequent write() fails fast with IOException('write has error. bs : pre upload obs[<key>] has error.'). This is a secondary, deliberately uninformative error — the real failure was logged when hasException was set. Its purpose is to stop the write path from piling more data onto a stream that is already known-broken.","triggerScenarios":"A background block upload (multipart part) failed earlier — network reset, 401/403 expired credentials, 5xx from OBS, out-of-memory in the buffering block factory — and the caller ignores the first exception and keeps writing; async failure happens between two write() calls, so the next write() sees hasException==true; previous flush() swallowed an exception in an outer wrapper.","commonSituations":"Long-running Spark/Hive writes where a transient network blip failed one part upload; AK/SK revocation mid-job; executors with tight memory causing block allocation failures; callers that catch-and-continue around write loops.","solutions":["Scroll UP in the logs to the first exception for this key — that root cause (auth, network, quota) is what to fix; this exception is only the messenger.","Close/abort the stream and fail the task; then retry the whole write from the last committed point (multipart parts already uploaded are discarded on abort).","If root cause is transient (5xx, connection reset), enable/verify fs.obs.attempts and retry policy on the client and retry the job.","If root cause is credentials, refresh AK/SK or session token before the retry; expired temporary credentials are the most common cause in long jobs."],"exampleFix":"// before\ntry { out.write(buf, 0, n); } catch (IOException e) { log.warn(\"write failed, keep going\"); }\nout.write(more); // -> 'pre upload obs[...] has error'\n\n// after\ntry {\n  out.write(buf, 0, n);\n} catch (IOException e) {\n  quietlyAbort(out); // ((OBSBlockOutputStream) out.getWrappedStream()).abort() or close quietly\n  throw new RuntimeException(\"OBS write failed for \" + path, e); // fail task, retry job\n}","handlingStrategy":"try-catch","validationCode":"// no public isFailed() exists; track the first failure in your wrapper\nclass SafeObsWriter {\n  private volatile boolean failed;\n  void write(FSDataOutputStream out, byte[] b, int off, int len) throws IOException {\n    if (failed) throw new IllegalStateException(\"stream already failed\");\n    try { out.write(b, off, len); }\n    catch (IOException e) { failed = true; throw e; }\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  out.write(buf, 0, n);\n} catch (IOException e) {\n  if (String.valueOf(e.getMessage()).contains(\"pre upload obs[\")) {\n    abortQuietly(out);\n    throw new RuntimeException(\"OBS stream poisoned by earlier failure — see first exception in logs\", e);\n  }\n  throw e;\n}","preventionTips":["Never catch-and-continue around write(); fail the task on the FIRST exception.","Rotate temporary credentials before expiry in long jobs.","Alert on the first upload failure, not on the cascade of 'pre upload obs[...] has error' messages.","Wrap streams in a failing-fast proxy that remembers the first error."],"tags":["obs","huaweicloud","output-stream","fail-fast","secondary-error"],"backgroundTag":"broken-output-stream","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}