apache/druid · error · DruidException

Got error[ ] from S3 when uploading segment of size[%,d]…

Error message

Got error[%s] from S3 when uploading segment of size[%,d] bytes. This typically happens when segment size is above 5GB. Try reducing your segment size by lowering the target number of rows per segment.

What it means

S3 rejected the upload of a zipped segment during deep storage push. S3DataSegmentPusher catches S3Exception from the put and converts it via handlePushServiceException with a hint that segments above the 5 GB single-PUT limit are the usual cause. The original S3 error code/message is embedded via %s.

Solutions

Reduce segment size: lower target rows per segment (e.g. use best-effort partitioning / smaller maxRowsPerSegment) so the zip stays under 5 GB.
Read the embedded S3 error code — if not size-related, fix permissions/bucket config or retry on throttling.
Enable Compaction with smaller segment granularity/rows to shrink existing oversized segments.

Example fix

// before
"maxRowsPerSegment": 10000000
// after
"maxRowsPerSegment": 3000000

Defensive patterns

Strategy: validation

Validate before calling

if (indexSize >= 5L * 1024 * 1024 * 1024) {
  throw new IllegalStateException("segment zip " + indexSize + " bytes exceeds S3 5GB PUT limit; reduce maxRowsPerSegment");
}

Try / catch

try {
  pusher.push(segmentDir, segmentDescriptor);
} catch (Exception e) {
  if (String.valueOf(e).contains("Got error") && String.valueOf(e).contains("5GB")) {
    // shrink partitions and re-run task
  }
}

Prevention

Keep segments well under 5 GB (typical target 300-700 MB)
Set sensible maxRowsPerSegment/partition tuning
Verify S3 credentials and bucket permissions before bulk loads
Use compaction to split oversized segments

When it happens

Trigger: pushZip → S3 putObject of a segment zip larger than 5 GB, or any S3-side failure (permissions, bucket issues) during segment upload.

Common situations: Overly large segments from high target row counts or wide schemas; misconfigured S3 credentials/bucket policy; S3 throttling during bulk load.

Understand the failure class

Background: payload too large / request exceeds maximum size: why libraries cap bytes and how to fix oversize payloads — this error's family across 50 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/541a35b28bd17bf5. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentPusher.java:98

    final File zipOutFile = Files.createTempFile("druid", "index.zip").toFile();
    try {
      final long indexSize = CompressionUtils.zip(indexFilesDir, zipOutFile);

      final DataSegment outSegment = baseSegment.withSize(indexSize)
                                                .withLoadSpec(makeLoadSpec(config.getBucket(), s3Path))
                                                .withBinaryVersion(SegmentUtils.getVersionFromDir(indexFilesDir));

      try {
        return S3Utils.retryS3Operation(
            () -> {
              S3Utils.uploadFileIfPossible(s3Client, config.getDisableAcl(), config.getBucket(), s3Path, zipOutFile);

              return outSegment;
            }
        );
      }
      catch (S3Exception e) {
        throw handlePushServiceException(e, indexSize);
      }
      catch (Exception e) {
        throw new RuntimeException(e);
      }
    }
    finally {
      log.debug("Deleting temporary cached index.zip");
      zipOutFile.delete();
    }
  }

  private DataSegment pushNoZip(File indexFilesDir, DataSegment baseSegment, String s3Path) throws IOException
  {
    final File[] files = indexFilesDir.listFiles();
    if (files == null) {
      throw new IOE("Cannot list directory [%s]", indexFilesDir);
    }

View on GitHub (pinned to 9b90983fd2)