{"record":{"id":"f9b07b1ab366bcb2","repo":"apache/hadoop","slug":"upload-failed-for-s-reason-s","errorCode":null,"errorMessage":"Upload failed for '%s'. reason=%s","messagePattern":"Upload failed for '(.+?)'\\. reason=(.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"critical","filePath":"hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientWriteChannel.java","lineNumber":102,"sourceCode":"    //TODO: Enable KMS and checksum\n    return blobWriteOptions.toArray(new BlobWriteOption[blobWriteOptions.size()]);\n  }\n\n  @Override\n  public boolean isOpen() {\n    return writableByteChannel != null && writableByteChannel.isOpen();\n  }\n\n  @Override\n  public void close() throws IOException {\n    try {\n      if (!isOpen()) {\n        return;\n      }\n\n      writableByteChannel.close();\n    } catch (Exception e) {\n      throw new IOException(\n          String.format(\"Upload failed for '%s'. reason=%s\", resourceId, e.getMessage()), e);\n    } finally {\n      writableByteChannel = null;\n    }\n  }\n\n  private int writeInternal(final ByteBuffer byteBuffer) throws IOException {\n    int bytesWritten = writableByteChannel.write(byteBuffer);\n    LOG.trace(\"{} bytes were written out of provided buffer of capacity {}\", bytesWritten,\n        byteBuffer.limit());\n    return bytesWritten;\n  }\n\n  @Override\n  public int write(final ByteBuffer src) throws IOException {\n    return writeInternal(src);\n  }\n}","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientWriteChannel.java#L84-L120","documentation":"GoogleCloudStorageClientWriteChannel.close() finalizes the upload: any exception from writableByteChannel.close() — the final commit of a resumable upload — is wrapped into IOException \"Upload failed for '<resourceId>'. reason=<message>\". Until close() succeeds the object typically does not exist or is incomplete; the data is not committed. The underlying reason (auth expiry, 5xx, checksum mismatch, connection drop) is both embedded in the message and attached as cause.","triggerScenarios":"Closing the write channel when the resumable-upload session fails to finalize: expired OAuth token or service-account key mid-job, GCS 5xx at commit time, CRC32C/MD5 mismatch detected at finalize, connection reset, or storage-client session limits on very large uploads.","commonSituations":"Spark/Hadoop task output committed at task end failing during credential rotation; long uploads dropped by networks at the final PUT; checksum mismatches from corrupted in-memory buffers; retries after partial sessions.","solutions":["Retry the whole write to a fresh object (or temp name then rename) — a session that fails at finalize generally cannot be resumed through the same channel.","Read the reason/cause: 401/403 -> refresh credentials/keys before the job; 5xx/429 -> backoff and retry; checksum mismatch -> verify the data source and buffers, then rewrite.","Write to a temporary name and atomically rename/copy on success so failed finalizes never leave partial outputs visible.","Upgrade the connector and storage client; align upload chunk size (fs.gs.outputstream.upload.chunk.size) for reliability on your network."],"exampleFix":"// before\ntry (WritableByteChannel out = gcs.create(id, opts)) {\n  writeAll(out, data);\n} // close() throws 'Upload failed' -> partial/invisible output\n\n// after: temp name + atomic publish, retried on failure\nfor (int attempt = 1; attempt <= 3; attempt++) {\n  try (WritableByteChannel out = gcs.create(tmpId, opts)) {\n    writeAll(out, data);\n  }\n  gcs.rename(tmpId, finalId); // publish\n  break;\n} // catch IOException -> next attempt","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) { // from close()\n  String msg = e.getMessage(); // contains 'Upload failed' + reason\n  boolean auth = msg.contains(\"401\") || msg.contains(\"403\");\n  if (auth) refreshCredentials();\n  retryWholeWriteToTempThenRename(); // never trust a partially finalized upload\n}","preventionTips":["Always write to a temp object and atomically publish (rename/copy) after a successful close.","Refresh service-account credentials before long jobs; watch token expiry windows.","Log the embedded reason= text — it distinguishes auth, 5xx, and checksum failures."],"tags":["gcs","upload","write-channel","resumable-upload","commit-failed"],"backgroundTag":"upload-commit-failed","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-23T01:17:44.959Z"}