What causes "File [%s] end marker not in expected location" in apache/druid?

Regenerate the frame file from the producing job; the footer layout cannot be repaired in place.

apache/druid · error · IOException

File [%s] end marker not in expected location

Error message

File [%s] end marker not in expected location

What it means

FrameFileFooter's constructor validates that the footer region begins with MARKER_NO_MORE_FRAMES, marking the end of frame data. If the marker byte is missing, the footer offset computed from the trailer does not land where the writer placed it, so the file layout is corrupt or was written by an incompatible version.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/file/FrameFileFooter.java:57

  private final long frameFileLength;

  public FrameFileFooter(Memory footerMemory, long frameFileLength) throws IOE
  {
    this.footerMemory = footerMemory;
    this.frameFileLength = frameFileLength;

    Memory trailer = footerMemory.region(
        footerMemory.getCapacity() - FrameFileWriter.TRAILER_LENGTH,
        FrameFileWriter.TRAILER_LENGTH,
        ByteOrder.LITTLE_ENDIAN
    );
    this.numFrames = trailer.getInt(0);
    this.numPartitions = trailer.getInt(Integer.BYTES);
    int length = trailer.getInt(Integer.BYTES * 2L);
    int expectedFooterChecksum = trailer.getInt(Integer.BYTES * 3L);
    // Verify footer begins with MARKER_NO_MORE_FRAMES.
    if (footerMemory.getByte(0) != FrameFileWriter.MARKER_NO_MORE_FRAMES) {
      throw new IOE("File [%s] end marker not in expected location", "file");
    }

    // Verify footer checksum.
    final int actualChecksum =
        (int) footerMemory.xxHash64(0, footerMemory.getCapacity() - Integer.BYTES, FrameFileWriter.CHECKSUM_SEED);

    if (expectedFooterChecksum != actualChecksum) {
      throw new ISE("Expected footer checksum did not match actual checksum. Corrupt or truncated file?");
    }

    // Verify footer length.
    if (length != FrameFileWriter.footerLength(numFrames, numPartitions)) {
      throw new ISE("Expected footer length did not match actual footer length. Corrupt or truncated file?");
    }
  }

  /**
   * First frame of a given partition. Partitions beyond {@link #getNumPartitions()} are treated as empty: if provided,

View on GitHub (pinned to 9b90983fd2)

Solutions

Regenerate the frame file from the producing job; the footer layout cannot be repaired in place.
Ensure the Druid version reading the file matches the version that wrote it.
Verify the file was not truncated in transit (checksum / size comparison against the producer's output).
Do not read frame files until the writer has fully closed them.

Example fix

// before
FrameFileFooter footer = FrameFile.open(maybeTruncatedFile, maxMmapSize).footer();
// after
if (!verifyChecksum(maybeTruncatedFile, producerChecksum)) {
  throw new IllegalStateException("Frame file corrupted in transit: " + maybeTruncatedFile);
}
FrameFileFooter footer = FrameFile.open(maybeTruncatedFile, maxMmapSize).footer();

Defensive patterns

Strategy: try-catch

Validate before calling

if (!checksumMatches(file, producerManifest.getChecksum())) {
  throw new IllegalStateException("Frame file corrupted before read: " + file);
}

Try / catch

try {
  FrameFile.open(file, maxMmapSize);
} catch (IOException e) {
  // 'end marker not in expected location': layout corrupt or version mismatch
  throw new UncheckedIOException("Frame file footer invalid, regenerating: " + file, e);
}

Prevention

Match Druid versions between writer and reader.
Store and verify a checksum alongside each produced frame file.
Wait for producer task completion before consuming outputs.
Avoid storage layers that can serve partial/torn writes.

When it happens

Trigger: Constructing FrameFileFooter for a file whose footer start offset (derived from footerLength in the trailer) points at bytes that are not the end marker — truncated files, wrong footerLength, or corrupted footer region.

Common situations: A frame file was truncated so the footer sits at the wrong offset; mixing Druid versions where the frame file layout changed; on-disk corruption after writing; reading a snapshot of a file mid-write.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/fa5fd89416c9c9a5. Report an issue: GitHub.