apache/druid · error · IOException

File[%s] is too short (size[%,d])

Error message

File[%s] is too short (size[%,d])

What it means

FrameFile.open() validates the minimum length of a frame file: the magic bytes plus the trailer plus the no-more-frames marker byte. A file shorter than this cannot possibly contain a valid header and footer, so open fails with an IOE reporting the actual size. This guards against reading empty or truncated frame files.

Source

Thrown at processing/src/main/java/org/apache/druid/frame/file/FrameFile.java:198

  ) throws IOException
  {
    final EnumSet<Flag> flagSet = flags.length == 0 ? EnumSet.noneOf(Flag.class) : EnumSet.copyOf(Arrays.asList(flags));

    if (!file.exists()) {
      throw new FileNotFoundException(StringUtils.format("File[%s] not found", file));
    }

    // Closer for mmap that is shared across all references: either footer only (if file size is larger
    // Integer.MAX_VALUE) or entire file (if file size is smaller than, or equal to, Integer.MAX_VALUE).
    Closeable sharedMapCloser = null;

    try (final RandomAccessFile randomAccessFile = new RandomAccessFile(file, "r")) {
      final long fileLength = randomAccessFile.length();

      // Verify minimum file length.
      if (fileLength <
          FrameFileWriter.MAGIC.length + FrameFileWriter.TRAILER_LENGTH + Byte.BYTES /* MARKER_NO_MORE_FRAMES */) {
        throw new IOE("File[%s] is too short (size[%,d])", file, fileLength);
      }

      // Verify magic.
      final byte[] buf = new byte[FrameFileWriter.TRAILER_LENGTH /* Larger than FrameFileWriter.MAGIC */];
      final Memory bufMemory = Memory.wrap(buf, ByteOrder.LITTLE_ENDIAN);
      randomAccessFile.readFully(buf, 0, FrameFileWriter.MAGIC.length);

      if (!bufMemory.equalTo(0, Memory.wrap(FrameFileWriter.MAGIC), 0, FrameFileWriter.MAGIC.length)) {
        throw new IOE("File[%s] is not a frame file", file);
      }

      // Read number of frames and partitions.
      randomAccessFile.seek(fileLength - FrameFileWriter.TRAILER_LENGTH);
      randomAccessFile.readFully(buf, 0, FrameFileWriter.TRAILER_LENGTH);

      final int footerLength = bufMemory.getInt(Integer.BYTES * 2L);
      if (footerLength < 0) {
        throw new ISE("Negative-size footer. Corrupt or truncated file[%s]?", file);

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Regenerate the frame by re-running the stage/query that wrote it.
  2. Check free disk space and writer logs for incomplete flushes or crashes.
  3. Skip/ignore the corrupt partial file if the writer is idempotent and will rewrite it.
  4. Verify any transfer/copy step preserves full file size.

Example fix

// before
FrameFile.open(new File(partitionPath)); // File[...] is too short (size[0])

// after
if (new File(partitionPath).length() < 64) { // below MIN_FRAME_FILE_LENGTH
  throw new IllegalStateException("Incomplete frame partition, re-run stage: " + partitionPath);
}
FrameFile.open(new File(partitionPath));
Defensive patterns

Strategy: validation

Validate before calling

// Java: sanity-check size before opening
final long MIN_FRAME_FILE_SIZE = 64; // MAGIC + TRAILER + marker
if (frameFile.length() < MIN_FRAME_FILE_SIZE) {
  throw new IOException("Frame file incomplete, regenerating: " + frameFile);
}

Try / catch

try {
  FrameFile.open(file, flags);
} catch (IOException e) {
  if (e.getMessage() != null && e.getMessage().contains("is too short")) {
    throw new RetryableQueryException("Truncated frame file, re-run stage: " + file, e);
  }
  throw e;
}

Prevention

When it happens

Trigger: Opening a frame file whose size is smaller than FrameFileWriter.MAGIC.length + TRAILER_LENGTH + 1 bytes — zero-byte or partially written frame files, e.g. a writer crashed mid-flush or the file was truncated by storage.

Common situations: Durable storage truncated frame files (disk full, crash during write); downstream stage reading frames before the writer finished flushing; copying frames with size limits (e.g. tools that cap file size).

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/9e6543e4cfe01734. Report an issue: GitHub.