apache/druid · error · IllegalStateException

Expected footer length did not match actual footer length. C

Error message

Expected footer length did not match actual footer length. Corrupt or truncated file?

What it means

The footer records numFrames and numPartitions; the constructor recomputes the expected footer length with FrameFileWriter.footerLength and compares it to the footerLength stored in the trailer. A mismatch means the footer's declared structure does not match its declared size, indicating a corrupt or inconsistent file.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/file/FrameFileFooter.java:70

    this.numPartitions = trailer.getInt(Integer.BYTES);
    int length = trailer.getInt(Integer.BYTES * 2L);
    int expectedFooterChecksum = trailer.getInt(Integer.BYTES * 3L);
    // Verify footer begins with MARKER_NO_MORE_FRAMES.
    if (footerMemory.getByte(0) != FrameFileWriter.MARKER_NO_MORE_FRAMES) {
      throw new IOE("File [%s] end marker not in expected location", "file");
    }

    // Verify footer checksum.
    final int actualChecksum =
        (int) footerMemory.xxHash64(0, footerMemory.getCapacity() - Integer.BYTES, FrameFileWriter.CHECKSUM_SEED);

    if (expectedFooterChecksum != actualChecksum) {
      throw new ISE("Expected footer checksum did not match actual checksum. Corrupt or truncated file?");
    }

    // Verify footer length.
    if (length != FrameFileWriter.footerLength(numFrames, numPartitions)) {
      throw new ISE("Expected footer length did not match actual footer length. Corrupt or truncated file?");
    }
  }

  /**
   * First frame of a given partition. Partitions beyond {@link #getNumPartitions()} are treated as empty: if provided,
   * this method returns {@link #getNumFrames()}.
   */
  public int getPartitionStartFrame(final int partition)
  {
    if (partition < 0) {
      throw new IAE("Partition [%,d] out of bounds", partition);
    } else if (partition >= numPartitions) {
      // Frame might not have every partition, if some are empty.
      return numFrames;
    } else {
      final long partitionStartFrameLocation =
          footerMemory.getCapacity()
          - FrameFileWriter.TRAILER_LENGTH

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Regenerate the frame file; structural inconsistency in the footer is not repairable in place.
  2. Check the producing task's logs for crash/restart during file writes and eliminate double-writing.
  3. Ensure reader and writer Druid versions are compatible.
  4. Verify storage/disk integrity if this happens repeatedly on the same volume.

Example fix

// before
FrameFile.open(mergedFile, maxMmapSize); // merged from partial files, inconsistent footer
// after
File complete = jobOutput.getSingleCompleteFile(); // never stitch partial outputs
FrameFile.open(complete, maxMmapSize);
Defensive patterns

Strategy: try-catch

Try / catch

try {
  FrameFile.open(file, maxMmapSize);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("footer length did not match")) {
    throw new IllegalStateException("Frame file structurally inconsistent, rebuild: " + file, e);
  }
  throw e;
}

Prevention

When it happens

Trigger: Opening a frame file whose trailer footerLength disagrees with footerLength(numFrames, numPartitions) computed from the trailer — e.g. the trailer was overwritten, or the file mixes bytes from two different writer runs.

Common situations: Overlapping writes from a crashed and restarted task; corrupted disk sectors; files assembled incorrectly by external tooling; mixing Druid versions with different footer layouts.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/cb4cec1f4a98c8c3. Report an issue: GitHub.