{"record":{"id":"6d9c35c563ee59a1","repo":"apache/druid","slug":"oversize-footer-corrupt-or-truncated-file-s","errorCode":null,"errorMessage":"Oversize footer. Corrupt or truncated file[%s]?","messagePattern":"Oversize footer\\. Corrupt or truncated file\\[(.+?)\\]\\?","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/frame/file/FrameFile.java","lineNumber":218,"sourceCode":"\n      // Verify magic.\n      final byte[] buf = new byte[FrameFileWriter.TRAILER_LENGTH /* Larger than FrameFileWriter.MAGIC */];\n      final Memory bufMemory = Memory.wrap(buf, ByteOrder.LITTLE_ENDIAN);\n      randomAccessFile.readFully(buf, 0, FrameFileWriter.MAGIC.length);\n\n      if (!bufMemory.equalTo(0, Memory.wrap(FrameFileWriter.MAGIC), 0, FrameFileWriter.MAGIC.length)) {\n        throw new IOE(\"File[%s] is not a frame file\", file);\n      }\n\n      // Read number of frames and partitions.\n      randomAccessFile.seek(fileLength - FrameFileWriter.TRAILER_LENGTH);\n      randomAccessFile.readFully(buf, 0, FrameFileWriter.TRAILER_LENGTH);\n\n      final int footerLength = bufMemory.getInt(Integer.BYTES * 2L);\n      if (footerLength < 0) {\n        throw new ISE(\"Negative-size footer. Corrupt or truncated file[%s]?\", file);\n      } else if (footerLength > fileLength) {\n        throw new ISE(\"Oversize footer. Corrupt or truncated file[%s]?\", file);\n      }\n\n      final Memory wholeFileMemory;\n      final Memory footerMemory;\n\n      if (fileLength <= maxMmapSize) {\n        // Map entire file, use region for footer.\n        final MappedByteBufferHandler mapHandle = FileUtils.map(randomAccessFile, 0, fileLength);\n        sharedMapCloser = mapHandle;\n        wholeFileMemory = Memory.wrap(mapHandle.get(), ByteOrder.LITTLE_ENDIAN);\n\n        if (wholeFileMemory.getCapacity() != fileLength) {\n          // Check that the mapped file is the expected length. May differ if the file was updated while we're trying\n          // to map it.\n          throw new ISE(\"Memory map size does not match file size\");\n        }\n\n        footerMemory = wholeFileMemory.region(fileLength - footerLength, footerLength, ByteOrder.LITTLE_ENDIAN);","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/frame/file/FrameFile.java#L200-L236","documentation":"FrameFile.open rejects a footerLength larger than the total file length, which is structurally impossible for a valid frame file. Like the negative-footer check, this guards against corrupt or truncated input before mapping the footer region into memory.","triggerScenarios":"Opening a file where the trailer's footerLength field points beyond the file — caused by truncation that removed part of the file while the trailer remained, garbage trailer bytes, or reading a file from a different/incompatible writer version.","commonSituations":"A writer process crashed and the file on disk mixes data from two runs; a download was cut short but the tail bytes were kept; an older reader reads files from a newer writer with a changed trailer layout.","solutions":["Regenerate the frame file with a matching Druid version; ensure reader and writer versions agree on the frame file format.","Validate the file's integrity (size vs expected) before opening; re-run the ingestion/segment-building job.","Check for concurrent modification — do not open frame files while the writer is still running.","If files came from external storage, verify the download/copy completed (checksums, content-length)."],"exampleFix":"// before\nFrameFile.open(truncatedCopy, maxMmapSize);\n// after\nlong expected = expectedOutputSize(jobReport);\nif (truncatedCopy.length() != expected) {\n  // re-download or re-run producing job\n} else {\n  FrameFile.open(truncatedCopy, maxMmapSize);\n}","handlingStrategy":"validation","validationCode":"if (file.length() < FrameFileWriter.TRAILER_LENGTH || file.length() != expectedSizeFromProducer) {\n  throw new IllegalStateException(\"Frame file size mismatch, likely corrupt: \" + file);\n}","typeGuard":null,"tryCatchPattern":"try {\n  FrameFile.open(file, maxMmapSize);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"Oversize footer\")) {\n    // version mismatch or corruption: re-download or regenerate\n    throw new IllegalStateException(\"Frame file unusable: \" + file, e);\n  }\n  throw e;\n}","preventionTips":["Keep reader and writer Druid versions aligned for frame file interchange.","Never append to or edit frame files after production.","Re-download from source if a copy is suspected truncated.","Close the writer before any reader opens the file."],"tags":["io","corruption","truncation","druid"],"backgroundTag":"checksum-mismatch","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}