apache/druid · error · IllegalStateException

Oversize footer. Corrupt or truncated file[%s]?

Error message

Oversize footer. Corrupt or truncated file[%s]?

What it means

FrameFile.open rejects a footerLength larger than the total file length, which is structurally impossible for a valid frame file. Like the negative-footer check, this guards against corrupt or truncated input before mapping the footer region into memory.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/file/FrameFile.java:218

      // Verify magic.
      final byte[] buf = new byte[FrameFileWriter.TRAILER_LENGTH /* Larger than FrameFileWriter.MAGIC */];
      final Memory bufMemory = Memory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
      randomAccessFile.readFully(buf, 0, FrameFileWriter.MAGIC.length);

      if (!bufMemory.equalTo(0, Memory.wrap(FrameFileWriter.MAGIC), 0, FrameFileWriter.MAGIC.length)) {
        throw new IOE("File[%s] is not a frame file", file);
      }

      // Read number of frames and partitions.
      randomAccessFile.seek(fileLength - FrameFileWriter.TRAILER_LENGTH);
      randomAccessFile.readFully(buf, 0, FrameFileWriter.TRAILER_LENGTH);

      final int footerLength = bufMemory.getInt(Integer.BYTES * 2L);
      if (footerLength < 0) {
        throw new ISE("Negative-size footer. Corrupt or truncated file[%s]?", file);
      } else if (footerLength > fileLength) {
        throw new ISE("Oversize footer. Corrupt or truncated file[%s]?", file);
      }

      final Memory wholeFileMemory;
      final Memory footerMemory;

      if (fileLength <= maxMmapSize) {
        // Map entire file, use region for footer.
        final MappedByteBufferHandler mapHandle = FileUtils.map(randomAccessFile, 0, fileLength);
        sharedMapCloser = mapHandle;
        wholeFileMemory = Memory.wrap(mapHandle.get(), ByteOrder.LITTLE_ENDIAN);

        if (wholeFileMemory.getCapacity() != fileLength) {
          // Check that the mapped file is the expected length. May differ if the file was updated while we're trying
          // to map it.
          throw new ISE("Memory map size does not match file size");
        }

        footerMemory = wholeFileMemory.region(fileLength - footerLength, footerLength, ByteOrder.LITTLE_ENDIAN);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Regenerate the frame file with a matching Druid version; ensure reader and writer versions agree on the frame file format.
2. Validate the file's integrity (size vs expected) before opening; re-run the ingestion/segment-building job.
3. Check for concurrent modification — do not open frame files while the writer is still running.
4. If files came from external storage, verify the download/copy completed (checksums, content-length).

Example fix

// before
FrameFile.open(truncatedCopy, maxMmapSize);
// after
long expected = expectedOutputSize(jobReport);
if (truncatedCopy.length() != expected) {
  // re-download or re-run producing job
} else {
  FrameFile.open(truncatedCopy, maxMmapSize);
}

if (file.length() < FrameFileWriter.TRAILER_LENGTH || file.length() != expectedSizeFromProducer) {
  throw new IllegalStateException("Frame file size mismatch, likely corrupt: " + file);
}

try {
  FrameFile.open(file, maxMmapSize);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Oversize footer")) {
    // version mismatch or corruption: re-download or regenerate
    throw new IllegalStateException("Frame file unusable: " + file, e);
  }
  throw e;
}

When it happens

Trigger: Opening a file where the trailer's footerLength field points beyond the file — caused by truncation that removed part of the file while the trailer remained, garbage trailer bytes, or reading a file from a different/incompatible writer version.

Common situations: A writer process crashed and the file on disk mixes data from two runs; a download was cut short but the tail bytes were kept; an older reader reads files from a newer writer with a changed trailer layout.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Negative-size footer. Corrupt or truncated file[%s]?
• File[%s] is too short (size[%,d])
• File [%s] end marker not in expected location
• Expected footer checksum did not match actual checksum. Corr
• Expected footer length did not match actual footer length. C

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/6d9c35c563ee59a1. Report an issue: GitHub.