apache/druid · error · IllegalStateException

Negative-size footer. Corrupt or truncated file[%s]?

Error message

Negative-size footer. Corrupt or truncated file[%s]?

What it means

FrameFile.open reads the trailer's footerLength field; a negative value is impossible in a well-formed file, so it indicates corruption or truncation. The check catches files whose trailer bytes were read from the wrong offset or were garbage (e.g. a truncated or externally modified file).

Source

Thrown at processing/src/main/java/org/apache/druid/frame/file/FrameFile.java:216

        throw new IOE("File[%s] is too short (size[%,d])", file, fileLength);
      }

      // Verify magic.
      final byte[] buf = new byte[FrameFileWriter.TRAILER_LENGTH /* Larger than FrameFileWriter.MAGIC */];
      final Memory bufMemory = Memory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
      randomAccessFile.readFully(buf, 0, FrameFileWriter.MAGIC.length);

      if (!bufMemory.equalTo(0, Memory.wrap(FrameFileWriter.MAGIC), 0, FrameFileWriter.MAGIC.length)) {
        throw new IOE("File[%s] is not a frame file", file);
      }

      // Read number of frames and partitions.
      randomAccessFile.seek(fileLength - FrameFileWriter.TRAILER_LENGTH);
      randomAccessFile.readFully(buf, 0, FrameFileWriter.TRAILER_LENGTH);

      final int footerLength = bufMemory.getInt(Integer.BYTES * 2L);
      if (footerLength < 0) {
        throw new ISE("Negative-size footer. Corrupt or truncated file[%s]?", file);
      } else if (footerLength > fileLength) {
        throw new ISE("Oversize footer. Corrupt or truncated file[%s]?", file);
      }

      final Memory wholeFileMemory;
      final Memory footerMemory;

      if (fileLength <= maxMmapSize) {
        // Map entire file, use region for footer.
        final MappedByteBufferHandler mapHandle = FileUtils.map(randomAccessFile, 0, fileLength);
        sharedMapCloser = mapHandle;
        wholeFileMemory = Memory.wrap(mapHandle.get(), ByteOrder.LITTLE_ENDIAN);

        if (wholeFileMemory.getCapacity() != fileLength) {
          // Check that the mapped file is the expected length. May differ if the file was updated while we're trying
          // to map it.
          throw new ISE("Memory map size does not match file size");
        }

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Confirm the producing job completed successfully and finalized the file; re-run the writer job to regenerate the file.
2. Check whether the file is still being written — only read frame files after the writing task completes.
3. Verify the file was transferred in binary mode (no FTP/text-mode or encoding transformations).
4. Compare file size against the expected output of the producing job to detect truncation.

Example fix

// before
FrameFile.open(partialFile, maxMmapSize); // throws on truncated download
// after
if (isTaskComplete(jobHandle)) {
  FrameFile.open(partialFile, maxMmapSize);
} else {
  // wait for writer completion or re-run the job
}

if (file.length() != expectedSizeFromProducer) {
  throw new IllegalStateException("Frame file truncated: " + file);
}

try {
  FrameFile.open(file, maxMmapSize);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("Corrupt or truncated")) {
    regenerateFrameFile(file); // re-run producer task
  } else {
    throw e;
  }
}

When it happens

Trigger: Opening a frame file whose last TRAILER_LENGTH bytes do not encode a valid non-negative footerLength — typically a file truncated mid-write, padded, or written by an incompatible writer version.

Common situations: A task failed midway and the frame file was never finalized; the file was appended to or edited; transferring files in text/ASCII mode corrupted binary content; reading a file still being written by another process.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Oversize footer. Corrupt or truncated file[%s]?
• File[%s] is too short (size[%,d])
• File [%s] end marker not in expected location
• Expected footer checksum did not match actual checksum. Corr
• Expected footer length did not match actual footer length. C

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/7d44c01d1693c233. Report an issue: GitHub.