apache/hadoop · critical · IOException

Upload failed for '%s'. reason=%s

Error message

Upload failed for '%s'. reason=%s

What it means

GoogleCloudStorageClientWriteChannel.close() finalizes the upload: any exception from writableByteChannel.close() — the final commit of a resumable upload — is wrapped into IOException "Upload failed for '<resourceId>'. reason=<message>". Until close() succeeds the object typically does not exist or is incomplete; the data is not committed. The underlying reason (auth expiry, 5xx, checksum mismatch, connection drop) is both embedded in the message and attached as cause.

Source

Thrown at hadoop-cloud-storage-project/hadoop-gcp/src/main/java/org/apache/hadoop/fs/gs/GoogleCloudStorageClientWriteChannel.java:102

    //TODO: Enable KMS and checksum
    return blobWriteOptions.toArray(new BlobWriteOption[blobWriteOptions.size()]);
  }

  @Override
  public boolean isOpen() {
    return writableByteChannel != null && writableByteChannel.isOpen();
  }

  @Override
  public void close() throws IOException {
    try {
      if (!isOpen()) {
        return;
      }

      writableByteChannel.close();
    } catch (Exception e) {
      throw new IOException(
          String.format("Upload failed for '%s'. reason=%s", resourceId, e.getMessage()), e);
    } finally {
      writableByteChannel = null;
    }
  }

  private int writeInternal(final ByteBuffer byteBuffer) throws IOException {
    int bytesWritten = writableByteChannel.write(byteBuffer);
    LOG.trace("{} bytes were written out of provided buffer of capacity {}", bytesWritten,
        byteBuffer.limit());
    return bytesWritten;
  }

  @Override
  public int write(final ByteBuffer src) throws IOException {
    return writeInternal(src);
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the whole write to a fresh object (or temp name then rename) — a session that fails at finalize generally cannot be resumed through the same channel.
2. Read the reason/cause: 401/403 -> refresh credentials/keys before the job; 5xx/429 -> backoff and retry; checksum mismatch -> verify the data source and buffers, then rewrite.
3. Write to a temporary name and atomically rename/copy on success so failed finalizes never leave partial outputs visible.
4. Upgrade the connector and storage client; align upload chunk size (fs.gs.outputstream.upload.chunk.size) for reliability on your network.

Example fix

// before
try (WritableByteChannel out = gcs.create(id, opts)) {
  writeAll(out, data);
} // close() throws 'Upload failed' -> partial/invisible output

// after: temp name + atomic publish, retried on failure
for (int attempt = 1; attempt <= 3; attempt++) {
  try (WritableByteChannel out = gcs.create(tmpId, opts)) {
    writeAll(out, data);
  }
  gcs.rename(tmpId, finalId); // publish
  break;
} // catch IOException -> next attempt

catch (IOException e) { // from close()
  String msg = e.getMessage(); // contains 'Upload failed' + reason
  boolean auth = msg.contains("401") || msg.contains("403");
  if (auth) refreshCredentials();
  retryWholeWriteToTempThenRename(); // never trust a partially finalized upload
}

When it happens

Trigger: Closing the write channel when the resumable-upload session fails to finalize: expired OAuth token or service-account key mid-job, GCS 5xx at commit time, CRC32C/MD5 mismatch detected at finalize, connection reset, or storage-client session limits on very large uploads.

Common situations: Spark/Hadoop task output committed at task end failing during credential rotation; long uploads dropped by networks at the final PUT; checksum mismatches from corrupted in-memory buffers; retries after partial sessions.

Related errors

• Object %s already exists.
• Error accessing Bucket %s
• Error accessing %s
• Bucket doesn't match for source '%s' and destination '%s'!
• Listing '%s' failed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/f9b07b1ab366bcb2. Report an issue: GitHub.