{"record":{"id":"541a35b28bd17bf5","repo":"apache/druid","slug":"got-error-s-from-s3-when-uploading-segment-of-si","errorCode":null,"errorMessage":"Got error[%s] from S3 when uploading segment of size[%,d] bytes. This typically happens when segment size is above 5GB. Try reducing your segment size by lowering the target number of rows per segment.","messagePattern":"Got error\\[(.+?)\\] from S3 when uploading segment of size\\[%,d\\] bytes\\. This typically happens when segment size is above 5GB\\. Try reducing your segment size by lowering the target number of rows per segment\\.","errorType":"exception","errorClass":"DruidException","httpStatus":null,"severity":"error","filePath":"extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentPusher.java","lineNumber":98,"sourceCode":"    final File zipOutFile = Files.createTempFile(\"druid\", \"index.zip\").toFile();\n    try {\n      final long indexSize = CompressionUtils.zip(indexFilesDir, zipOutFile);\n\n      final DataSegment outSegment = baseSegment.withSize(indexSize)\n                                                .withLoadSpec(makeLoadSpec(config.getBucket(), s3Path))\n                                                .withBinaryVersion(SegmentUtils.getVersionFromDir(indexFilesDir));\n\n      try {\n        return S3Utils.retryS3Operation(\n            () -> {\n              S3Utils.uploadFileIfPossible(s3Client, config.getDisableAcl(), config.getBucket(), s3Path, zipOutFile);\n\n              return outSegment;\n            }\n        );\n      }\n      catch (S3Exception e) {\n        throw handlePushServiceException(e, indexSize);\n      }\n      catch (Exception e) {\n        throw new RuntimeException(e);\n      }\n    }\n    finally {\n      log.debug(\"Deleting temporary cached index.zip\");\n      zipOutFile.delete();\n    }\n  }\n\n  private DataSegment pushNoZip(File indexFilesDir, DataSegment baseSegment, String s3Path) throws IOException\n  {\n    final File[] files = indexFilesDir.listFiles();\n    if (files == null) {\n      throw new IOE(\"Cannot list directory [%s]\", indexFilesDir);\n    }\n","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentPusher.java#L80-L116","documentation":"S3 rejected the upload of a zipped segment during deep storage push. S3DataSegmentPusher catches S3Exception from the put and converts it via handlePushServiceException with a hint that segments above the 5 GB single-PUT limit are the usual cause. The original S3 error code/message is embedded via %s.","triggerScenarios":"pushZip → S3 putObject of a segment zip larger than 5 GB, or any S3-side failure (permissions, bucket issues) during segment upload.","commonSituations":"Overly large segments from high target row counts or wide schemas; misconfigured S3 credentials/bucket policy; S3 throttling during bulk load.","solutions":["Reduce segment size: lower target rows per segment (e.g. use best-effort partitioning / smaller maxRowsPerSegment) so the zip stays under 5 GB.","Read the embedded S3 error code — if not size-related, fix permissions/bucket config or retry on throttling.","Enable Compaction with smaller segment granularity/rows to shrink existing oversized segments."],"exampleFix":"// before\n\"maxRowsPerSegment\": 10000000\n// after\n\"maxRowsPerSegment\": 3000000","handlingStrategy":"validation","validationCode":"if (indexSize >= 5L * 1024 * 1024 * 1024) {\n  throw new IllegalStateException(\"segment zip \" + indexSize + \" bytes exceeds S3 5GB PUT limit; reduce maxRowsPerSegment\");\n}","typeGuard":null,"tryCatchPattern":"try {\n  pusher.push(segmentDir, segmentDescriptor);\n} catch (Exception e) {\n  if (String.valueOf(e).contains(\"Got error\") && String.valueOf(e).contains(\"5GB\")) {\n    // shrink partitions and re-run task\n  }\n}","preventionTips":["Keep segments well under 5 GB (typical target 300-700 MB)","Set sensible maxRowsPerSegment/partition tuning","Verify S3 credentials and bucket permissions before bulk loads","Use compaction to split oversized segments"],"tags":["s3","deep-storage","segment-push","payload-too-large"],"backgroundTag":"payload-too-large","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}