{"record":{"id":"fa5fd89416c9c9a5","repo":"apache/druid","slug":"file-s-end-marker-not-in-expected-location","errorCode":null,"errorMessage":"File [%s] end marker not in expected location","messagePattern":"File \\[(.+?)\\] end marker not in expected location","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"processing/src/main/java/org/apache/druid/frame/file/FrameFileFooter.java","lineNumber":57,"sourceCode":"  private final long frameFileLength;\n\n  public FrameFileFooter(Memory footerMemory, long frameFileLength) throws IOE\n  {\n    this.footerMemory = footerMemory;\n    this.frameFileLength = frameFileLength;\n\n    Memory trailer = footerMemory.region(\n        footerMemory.getCapacity() - FrameFileWriter.TRAILER_LENGTH,\n        FrameFileWriter.TRAILER_LENGTH,\n        ByteOrder.LITTLE_ENDIAN\n    );\n    this.numFrames = trailer.getInt(0);\n    this.numPartitions = trailer.getInt(Integer.BYTES);\n    int length = trailer.getInt(Integer.BYTES * 2L);\n    int expectedFooterChecksum = trailer.getInt(Integer.BYTES * 3L);\n    // Verify footer begins with MARKER_NO_MORE_FRAMES.\n    if (footerMemory.getByte(0) != FrameFileWriter.MARKER_NO_MORE_FRAMES) {\n      throw new IOE(\"File [%s] end marker not in expected location\", \"file\");\n    }\n\n    // Verify footer checksum.\n    final int actualChecksum =\n        (int) footerMemory.xxHash64(0, footerMemory.getCapacity() - Integer.BYTES, FrameFileWriter.CHECKSUM_SEED);\n\n    if (expectedFooterChecksum != actualChecksum) {\n      throw new ISE(\"Expected footer checksum did not match actual checksum. Corrupt or truncated file?\");\n    }\n\n    // Verify footer length.\n    if (length != FrameFileWriter.footerLength(numFrames, numPartitions)) {\n      throw new ISE(\"Expected footer length did not match actual footer length. Corrupt or truncated file?\");\n    }\n  }\n\n  /**\n   * First frame of a given partition. Partitions beyond {@link #getNumPartitions()} are treated as empty: if provided,","sourceCodeStart":39,"sourceCodeEnd":75,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/processing/src/main/java/org/apache/druid/frame/file/FrameFileFooter.java#L39-L75","documentation":"FrameFileFooter's constructor validates that the footer region begins with MARKER_NO_MORE_FRAMES, marking the end of frame data. If the marker byte is missing, the footer offset computed from the trailer does not land where the writer placed it, so the file layout is corrupt or was written by an incompatible version.","triggerScenarios":"Constructing FrameFileFooter for a file whose footer start offset (derived from footerLength in the trailer) points at bytes that are not the end marker — truncated files, wrong footerLength, or corrupted footer region.","commonSituations":"A frame file was truncated so the footer sits at the wrong offset; mixing Druid versions where the frame file layout changed; on-disk corruption after writing; reading a snapshot of a file mid-write.","solutions":["Regenerate the frame file from the producing job; the footer layout cannot be repaired in place.","Ensure the Druid version reading the file matches the version that wrote it.","Verify the file was not truncated in transit (checksum / size comparison against the producer's output).","Do not read frame files until the writer has fully closed them."],"exampleFix":"// before\nFrameFileFooter footer = FrameFile.open(maybeTruncatedFile, maxMmapSize).footer();\n// after\nif (!verifyChecksum(maybeTruncatedFile, producerChecksum)) {\n  throw new IllegalStateException(\"Frame file corrupted in transit: \" + maybeTruncatedFile);\n}\nFrameFileFooter footer = FrameFile.open(maybeTruncatedFile, maxMmapSize).footer();","handlingStrategy":"try-catch","validationCode":"if (!checksumMatches(file, producerManifest.getChecksum())) {\n  throw new IllegalStateException(\"Frame file corrupted before read: \" + file);\n}","typeGuard":null,"tryCatchPattern":"try {\n  FrameFile.open(file, maxMmapSize);\n} catch (IOException e) {\n  // 'end marker not in expected location': layout corrupt or version mismatch\n  throw new UncheckedIOException(\"Frame file footer invalid, regenerating: \" + file, e);\n}","preventionTips":["Match Druid versions between writer and reader.","Store and verify a checksum alongside each produced frame file.","Wait for producer task completion before consuming outputs.","Avoid storage layers that can serve partial/torn writes."],"tags":["io","corruption","druid"],"backgroundTag":"checksum-mismatch","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}