apache/druid · error · IllegalStateException

Expected footer checksum did not match actual checksum. Corr

Error message

Expected footer checksum did not match actual checksum. Corrupt or truncated file?

What it means

The footer carries an xxHash64 checksum of its own contents; FrameFileFooter recomputes it and compares. A mismatch means the footer bytes changed after writing — classic corruption/truncation — so the file cannot be trusted and reading is aborted.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/file/FrameFileFooter.java:65

        footerMemory.getCapacity() - FrameFileWriter.TRAILER_LENGTH,
        FrameFileWriter.TRAILER_LENGTH,
        ByteOrder.LITTLE_ENDIAN
    );
    this.numFrames = trailer.getInt(0);
    this.numPartitions = trailer.getInt(Integer.BYTES);
    int length = trailer.getInt(Integer.BYTES * 2L);
    int expectedFooterChecksum = trailer.getInt(Integer.BYTES * 3L);
    // Verify footer begins with MARKER_NO_MORE_FRAMES.
    if (footerMemory.getByte(0) != FrameFileWriter.MARKER_NO_MORE_FRAMES) {
      throw new IOE("File [%s] end marker not in expected location", "file");
    }

    // Verify footer checksum.
    final int actualChecksum =
        (int) footerMemory.xxHash64(0, footerMemory.getCapacity() - Integer.BYTES, FrameFileWriter.CHECKSUM_SEED);

    if (expectedFooterChecksum != actualChecksum) {
      throw new ISE("Expected footer checksum did not match actual checksum. Corrupt or truncated file?");
    }

    // Verify footer length.
    if (length != FrameFileWriter.footerLength(numFrames, numPartitions)) {
      throw new ISE("Expected footer length did not match actual footer length. Corrupt or truncated file?");
    }
  }

  /**
   * First frame of a given partition. Partitions beyond {@link #getNumPartitions()} are treated as empty: if provided,
   * this method returns {@link #getNumFrames()}.
   */
  public int getPartitionStartFrame(final int partition)
  {
    if (partition < 0) {
      throw new IAE("Partition [%,d] out of bounds", partition);
    } else if (partition >= numPartitions) {
      // Frame might not have every partition, if some are empty.

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Treat the file as unrecoverable and re-run the job that produced the frame file.
2. Verify storage health if corruption recurs (disk errors, NFS/S3 consistency issues).
3. Confirm the file transfer used binary mode and completed fully.
4. Ensure only one process writes the file and readers wait for task completion.

Example fix

// before
FrameFile.open(suspectFile, maxMmapSize); // ISE on checksum mismatch
// after
if (checksumMatches(suspectFile, manifest.checksum())) {
  FrameFile.open(suspectFile, maxMmapSize);
} else {
  regenerateFrameFile(manifest); // re-run producing task
}

// verify the file end-to-end before use
try (java.io.RandomAccessFile raf = new java.io.RandomAccessFile(file, "r")) {
  if (raf.length() != producerManifest.getSize()) throw new IllegalStateException("size mismatch");
}

try {
  FrameFile.open(file, maxMmapSize);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("checksum did not match")) {
    regenerateFrameFile(file); // corruption is not recoverable in place
  } else {
    throw e;
  }
}

When it happens

Trigger: Opening a frame file whose footer bytes were altered (bit rot, partial write, edit after production) so the stored checksum no longer matches the recomputed hash.

Common situations: Disk/storage corruption; incomplete file transfer or download; file modified after task completion; concurrency where a reader sees a partially flushed file from a crashed writer.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• File[%s] is too short (size[%,d])
• Negative-size footer. Corrupt or truncated file[%s]?
• Oversize footer. Corrupt or truncated file[%s]?
• File [%s] end marker not in expected location
• Expected footer length did not match actual footer length. C

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0f0ef448c2a3817a. Report an issue: GitHub.