{"record":{"id":"2592e69fe85b7cea","repo":"apache/druid","slug":"file-s-too-large-d-2592e6","errorCode":null,"errorMessage":"file[%s] too large [%,d]","messagePattern":"file\\[(.+?)\\] too large \\[%,d\\]","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/utils/CompressionUtils.java","lineNumber":277,"sourceCode":"   *\n   * @throws IOException\n   */\n  public static long zip(File directory, OutputStream out) throws IOException\n  {\n    if (!directory.isDirectory()) {\n      throw new IOE(\"directory[%s] is not a directory\", directory);\n    }\n\n    final ZipOutputStream zipOut = new ZipOutputStream(out);\n\n    long totalSize = 0;\n\n    // Sort entries to make life easier when writing streaming-decompression unit tests.\n    for (File file : Arrays.stream(directory.listFiles()).sorted().collect(Collectors.toList())) {\n      log.debug(\"Adding file[%s] with size[%,d].  Total size so far[%,d]\", file, file.length(), totalSize);\n      if (file.length() > Integer.MAX_VALUE) {\n        zipOut.finish();\n        throw new IOE(\"file[%s] too large [%,d]\", file, file.length());\n      }\n      zipOut.putNextEntry(new ZipEntry(file.getName()));\n      totalSize += Files.asByteSource(file).copyTo(zipOut);\n    }\n    zipOut.closeEntry();\n    // Workaround for http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/759aa847dcaf\n    zipOut.flush();\n    zipOut.finish();\n\n    return totalSize;\n  }\n\n  /**\n   * Compresses directory contents using LZ4 block compression with a simple archive format.\n   * Format: [file_count:4 bytes][file1_name_length:4][file1_name:bytes][file1_size:8][file1_data:bytes]...\n   *\n   * @param directory The directory whose contents should be compressed\n   * @param out The output stream to write compressed data to","sourceCodeStart":259,"sourceCodeEnd":295,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/utils/CompressionUtils.java#L259-L295","documentation":"While zipping, CompressionUtils.zip streams each entry through an in-memory buffer sized by Integer.MAX_VALUE; any single file larger than Integer.MAX_VALUE bytes (~2 GiB) cannot be handled and throws this IOException after finishing the zip stream. It is a hard size limit of the current implementation, not a ZIP-format limit.","triggerScenarios":"Calling CompressionUtils.zip on a directory that contains a file whose length exceeds Integer.MAX_VALUE (about 2.1 GB), e.g. a huge segment or log file inside the directory being archived.","commonSituations":"Archiving directories containing large raw dumps, unbounded logs, or oversized data files; environments where segment files grew past 2GB; older deep-storage push flows with big local staging dirs.","solutions":["Exclude or split files larger than Integer.MAX_VALUE before zipping","Use zip's size-aware behavior: check file.length() first and handle oversized files separately (e.g. compress individually with a stream compressor)","Move oversized files elsewhere and zip the remainder","Upgrade Druid if a newer implementation raised or removed the limit"],"exampleFix":"// before\nCompressionUtils.zip(dir, out); // throws if any file > 2GiB\n// after\nfor (File f : dir.listFiles()) {\n  if (f.length() > Integer.MAX_VALUE) {\n    throw new IllegalStateException(\"Split or exclude oversized file: \" + f);\n  }\n}\nCompressionUtils.zip(dir, out);","handlingStrategy":"validation","validationCode":"for (File f : dir.listFiles()) {\n  if (f.length() > Integer.MAX_VALUE) {\n    throw new IllegalStateException(\"File too large to zip: \" + f);\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  CompressionUtils.zip(dir, out);\n} catch (IOException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"too large\")) {\n    // exclude/split the oversized file and retry\n  } else throw e;\n}","preventionTips":["Check file.length() against Integer.MAX_VALUE before zipping","Exclude logs/dumps that can exceed 2GB from archive directories","Split or stream-compress very large files individually"],"tags":["java","ioexception","file-size-limit","compression"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}