{"record":{"id":"d88321f63a784c89","repo":"apache/druid","slug":"file-s-too-large-d","errorCode":null,"errorMessage":"File[%s] too large[%d]","messagePattern":"File\\[(.+?)\\] too large\\[(.+?)\\]","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/segment/IndexIO.java","lineNumber":232,"sourceCode":"\n  public QueryableIndex loadIndex(File inDir, boolean lazy, SegmentLazyLoadFailCallback loadFailed) throws IOException\n  {\n    final int version = SegmentUtils.getVersionFromDir(inDir);\n\n    final IndexLoader loader = indexLoaders.get(version);\n\n    if (loader != null) {\n      return loader.load(inDir, mapper, lazy, loadFailed);\n    } else {\n      throw new ISE(\"Unknown index version[%s]\", version);\n    }\n  }\n\n  public static void checkFileSize(File indexFile) throws IOException\n  {\n    final long fileSize = indexFile.length();\n    if (fileSize > Integer.MAX_VALUE) {\n      throw new IOE(\"File[%s] too large[%d]\", indexFile, fileSize);\n    }\n  }\n\n  interface IndexIOHandler\n  {\n    MMappedIndex mapDir(File inDir) throws IOException;\n  }\n\n  private static void validateRowValues(\n      RowPointer rp1,\n      IndexableAdapter adapter1,\n      RowPointer rp2,\n      IndexableAdapter adapter2\n  )\n  {\n    if (rp1.getTimestamp() != rp2.getTimestamp()) {\n      throw new SegmentValidationException(\n          \"Timestamp mismatch. Expected %d found %d\",","sourceCodeStart":214,"sourceCodeEnd":250,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/segment/IndexIO.java#L214-L250","documentation":"IndexIO.checkFileSize guards that a legacy (v8) segment index file can be memory-mapped or read into buffers indexed by int. If the file exceeds Integer.MAX_VALUE bytes, Druid throws IOE because its v8 storage code assumes 32-bit offsets. It is a pre-flight sanity check before loading old-format segments.","triggerScenarios":"Calling IndexIO.checkFileSize on any index.drd / segment file whose length() > 2,147,483,647 bytes, typically while mapping a legacy v8 segment directory.","commonSituations":"Very large legacy segments produced by long-retention ingestion jobs; v8 segments that grew beyond 2GB due to unbounded metric cardinality or long rollup windows.","solutions":["Reduce segment size by splitting the segment's interval into smaller shards and re-persisting","Upgrade the segment to v9+ format (which does not go through this 2GB int-offset path) via IndexMerger conversion","Re-ingest the data with a finer granularity or smaller maxRowsInMemory/partition settings"],"exampleFix":"// before\nIndexIO.checkFileSize(largeIndexFile);\n// after\nif (largeIndexFile.length() > Integer.MAX_VALUE) {\n  // split or convert the segment before mapping\n  throw new SegmentLoadingException(\"Segment exceeds 2GB v8 limit, re-merge into smaller shards\");\n}\nIndexIO.checkFileSize(largeIndexFile);","handlingStrategy":"validation","validationCode":"if (indexFile.length() > Integer.MAX_VALUE) {\n  throw new IllegalStateException(\"Segment file too large for v8 loader: \" + indexFile);\n}\nIndexIO.checkFileSize(indexFile);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Check file length before loading any legacy segment","Keep segment sizes under 2GB via partitioning/granularity settings","Prefer v9 segment loading paths which avoid int-offset limits"],"tags":["druid","segment","io","file-size"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}