apache/druid · error · IOException

file[ ] too large [%,d]

Error message

file[%s] too large [%,d]

What it means

While zipping, CompressionUtils.zip streams each entry through an in-memory buffer sized by Integer.MAX_VALUE; any single file larger than Integer.MAX_VALUE bytes (~2 GiB) cannot be handled and throws this IOException after finishing the zip stream. It is a hard size limit of the current implementation, not a ZIP-format limit.

Solutions

1. Exclude or split files larger than Integer.MAX_VALUE before zipping
2. Use zip's size-aware behavior: check file.length() first and handle oversized files separately (e.g. compress individually with a stream compressor)
3. Move oversized files elsewhere and zip the remainder
4. Upgrade Druid if a newer implementation raised or removed the limit

Example fix

// before
CompressionUtils.zip(dir, out); // throws if any file > 2GiB
// after
for (File f : dir.listFiles()) {
  if (f.length() > Integer.MAX_VALUE) {
    throw new IllegalStateException("Split or exclude oversized file: " + f);
  }
}
CompressionUtils.zip(dir, out);

for (File f : dir.listFiles()) {
  if (f.length() > Integer.MAX_VALUE) {
    throw new IllegalStateException("File too large to zip: " + f);
  }
}

try {
  CompressionUtils.zip(dir, out);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("too large")) {
    // exclude/split the oversized file and retry
  } else throw e;
}

When it happens

Trigger: Calling CompressionUtils.zip on a directory that contains a file whose length exceeds Integer.MAX_VALUE (about 2.1 GB), e.g. a huge segment or log file inside the directory being archived.

Common situations: Archiving directories containing large raw dumps, unbounded logs, or oversized data files; environments where segment files grew past 2GB; older deep-storage push flows with big local staging dirs.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• directory[ ] is not a directory
• Directory compression not supported for
• Directory decompression not supported for
• NONE compression strategy shouldn't use any compressor
• NONE compression strategy shouldn't use any decompressor

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/2592e69fe85b7cea. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/utils/CompressionUtils.java:277

   *
   * @throws IOException
   */
  public static long zip(File directory, OutputStream out) throws IOException
  {
    if (!directory.isDirectory()) {
      throw new IOE("directory[%s] is not a directory", directory);
    }

    final ZipOutputStream zipOut = new ZipOutputStream(out);

    long totalSize = 0;

    // Sort entries to make life easier when writing streaming-decompression unit tests.
    for (File file : Arrays.stream(directory.listFiles()).sorted().collect(Collectors.toList())) {
      log.debug("Adding file[%s] with size[%,d].  Total size so far[%,d]", file, file.length(), totalSize);
      if (file.length() > Integer.MAX_VALUE) {
        zipOut.finish();
        throw new IOE("file[%s] too large [%,d]", file, file.length());
      }
      zipOut.putNextEntry(new ZipEntry(file.getName()));
      totalSize += Files.asByteSource(file).copyTo(zipOut);
    }
    zipOut.closeEntry();
    // Workaround for http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/759aa847dcaf
    zipOut.flush();
    zipOut.finish();

    return totalSize;
  }

  /**
   * Compresses directory contents using LZ4 block compression with a simple archive format.
   * Format: [file_count:4 bytes][file1_name_length:4][file1_name:bytes][file1_size:8][file1_data:bytes]...
   *
   * @param directory The directory whose contents should be compressed
   * @param out The output stream to write compressed data to

View on GitHub (pinned to 9b90983fd2)