apache/druid · critical · IOException

Header file[ ] is [ ] bytes on disk but its metadata…

Error message

Header file[%s] is [%d] bytes on disk but its metadata describes [%d] bytes (header[%d] plus bitmap[%d]); treating it as corrupt

What it means

PartialSegmentFileMapperV10 validates that a header file's on-disk size exactly matches the size recorded in its metadata (header bytes plus bitmap bytes). If they differ, the header file is treated as corrupt and an IOException is thrown so the segment file will not be trusted. This protects readers from using a truncated or padded header that would misalign bitmap and metadata reads.

Solutions

1. Re-download or re-copy the segment from a healthy replica so the header file matches its metadata size
2. Re-ingest or re-index the affected segment from source data to regenerate a complete header file
3. Check for disk-full or interrupted writes on the historical node's segment cache directory and clean up corrupt entries (remove the segment dir so it is re-fetched)

Example fix

// no caller code fix; recover the data
# before: corrupt segment dir in cache
# after: remove and re-fetch
rm -rf /var/druid/segment-cache/<segment_id>

// Java: verify header file size before using the segment
long expected = metadata.getHeaderSize() + metadata.getBitmapSize();
if (new File(headerPath).length() != expected) {
  throw new IOException("corrupt header: " + headerPath);
}

try {
  openSegmentFile(segmentDir);
} catch (IOException e) {
  if (e.getMessage().contains("treating it as corrupt")) {
    deleteSegmentFromCache(segmentDir); // force re-fetch from deep storage
    retryDownload(segmentId);
  } else { throw e; }
}

When it happens

Trigger: Reading a segment file whose <name>.hdr companion file was truncated by a partial write, disk-full event, interrupted download, or manual copy while verifyPersistedHeaderLength computes headerFileSize(result) != headerFile.length().

Common situations: Segments copied between nodes with rsync/scp interrupted mid-transfer; crash during persist leaving a partial header; restoring from an incomplete backup; disk-full conditions during segment write.

Understand the failure class

Background: Checksum mismatch errors: "checksum verification failed", "digest mismatch", "expected vs actual checksum" — what they mean and how to fix them — this error's family across 41 libraries.

Related errors

• Corrupt frame file: frame [%,d] location out of range
• Expected [%,d] bytes, only saw [%,d], potential corruption?
• Expected footer checksum did not match actual checksum…
• Expected footer length did not match actual footer length…
• File [ ] end marker not in expected location

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/50ef79716d238d15. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/file/PartialSegmentFileMapperV10.java:1421

    return mapUninterruptibly(() -> {
      try (RandomAccessFile raf = new RandomAccessFile(headerFile, "rw");
           FileChannel channel = raf.getChannel()) {
        return channel.map(FileChannel.MapMode.READ_WRITE, result.getHeaderSize(), numBitmapBytes);
      }
    });
  }

  /**
   * Corruption check for a header file restored from a previous session: its length must be exactly the header plus
   * one bit per internal file, since {@link #fetchAndPersistHeader} only ever publishes both regions together.
   */
  private static void verifyPersistedHeaderLength(File headerFile, SegmentFileMetadataReader.Result result)
      throws IOException
  {
    final long expectedSize = headerFileSize(result);
    final long actualSize = headerFile.length();
    if (actualSize != expectedSize) {
      throw new IOException(
          StringUtils.format(
              "Header file[%s] is [%d] bytes on disk but its metadata describes [%d] bytes (header[%d] plus "
              + "bitmap[%d]); treating it as corrupt",
              headerFile,
              actualSize,
              expectedSize,
              result.getHeaderSize(),
              numBitmapBytes(result.getMetadata())
          )
      );
    }
  }

  /**
   * Establish a memory mapping, shielding it from the calling thread's interrupt status.
   * <p>
   * {@link FileChannel#map} is an interruptible channel operation: an interrupt (a canceled query, a stage tearing
   * down, {@code shutdownNow} on the processing pool) closes the channel mid-call and surfaces as

View on GitHub (pinned to 9b90983fd2)