apache/druid · error · IOException

File[%s] too large[%d]

Error message

File[%s] too large[%d]

What it means

IndexIO.checkFileSize guards that a legacy (v8) segment index file can be memory-mapped or read into buffers indexed by int. If the file exceeds Integer.MAX_VALUE bytes, Druid throws IOE because its v8 storage code assumes 32-bit offsets. It is a pre-flight sanity check before loading old-format segments.

Source

Thrown at processing/src/main/java/org/apache/druid/segment/IndexIO.java:232

  public QueryableIndex loadIndex(File inDir, boolean lazy, SegmentLazyLoadFailCallback loadFailed) throws IOException
  {
    final int version = SegmentUtils.getVersionFromDir(inDir);

    final IndexLoader loader = indexLoaders.get(version);

    if (loader != null) {
      return loader.load(inDir, mapper, lazy, loadFailed);
    } else {
      throw new ISE("Unknown index version[%s]", version);
    }
  }

  public static void checkFileSize(File indexFile) throws IOException
  {
    final long fileSize = indexFile.length();
    if (fileSize > Integer.MAX_VALUE) {
      throw new IOE("File[%s] too large[%d]", indexFile, fileSize);
    }
  }

  interface IndexIOHandler
  {
    MMappedIndex mapDir(File inDir) throws IOException;
  }

  private static void validateRowValues(
      RowPointer rp1,
      IndexableAdapter adapter1,
      RowPointer rp2,
      IndexableAdapter adapter2
  )
  {
    if (rp1.getTimestamp() != rp2.getTimestamp()) {
      throw new SegmentValidationException(
          "Timestamp mismatch. Expected %d found %d",

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Reduce segment size by splitting the segment's interval into smaller shards and re-persisting
2. Upgrade the segment to v9+ format (which does not go through this 2GB int-offset path) via IndexMerger conversion
3. Re-ingest the data with a finer granularity or smaller maxRowsInMemory/partition settings

Example fix

// before
IndexIO.checkFileSize(largeIndexFile);
// after
if (largeIndexFile.length() > Integer.MAX_VALUE) {
  // split or convert the segment before mapping
  throw new SegmentLoadingException("Segment exceeds 2GB v8 limit, re-merge into smaller shards");
}
IndexIO.checkFileSize(largeIndexFile);

if (indexFile.length() > Integer.MAX_VALUE) {
  throw new IllegalStateException("Segment file too large for v8 loader: " + indexFile);
}
IndexIO.checkFileSize(indexFile);

When it happens

Trigger: Calling IndexIO.checkFileSize on any index.drd / segment file whose length() > 2,147,483,647 bytes, typically while mapping a legacy v8 segment directory.

Common situations: Very large legacy segments produced by long-retention ingestion jobs; v8 segments that grew beyond 2GB due to unbounded metric cardinality or long rollup windows.

Understand the failure class

Background: "File too large" / "file size exceeds limit" errors: why libraries cap file sizes and how to fix them — this error's family across 46 libraries.

Related errors

• Failed to deserialize V%s column.
• Failed to deserialize V%s column.
• Failed to deserialize V%s column.
• Failed to deserialize V%s column.
• Input stream is null

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/d88321f63a784c89. Report an issue: GitHub.