apache/hadoop · error · IOException

File {} had invalid length: {}

Error message

File {} had invalid length: {}

What it means

BestEffortLongFile persists one 64-bit long in raw binary form (e.g., the QJM Journal server's committed transaction id, see Journal.java:210). A missing or empty file is fine (default value is used), but on the first get()/set() the lazy loader throws this IOException when the file exists with any length other than exactly 8 bytes - a partial write cannot be interpreted as any valid long.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/util/BestEffortLongFile.java:92

    value = newVal;
  }
  
  private void lazyOpen() throws IOException {
    if (ch != null) {
      return;
    }

    // Load current value.
    byte[] data = null;
    try {
      data = Files.toByteArray(file);
    } catch (FileNotFoundException fnfe) {
      // Expected - this will use default value.
    }

    if (data != null && data.length != 0) {
      if (data.length != Longs.BYTES) {
        throw new IOException("File " + file + " had invalid length: " +
            data.length);
      }
      value = Longs.fromByteArray(data);
    } else {
      value = defaultVal;
    }
    
    // Now open file for future writes.
    RandomAccessFile raf = new RandomAccessFile(file, "rw");
    try {
      ch = raf.getChannel();
    } finally {
      if (ch == null) {
        IOUtils.closeStream(raf);
      }
    }
  }

View on GitHub (pinned to 2add963021)

Solutions

  1. Confirm with 'ls -l <file>' - any size other than 0 or 8 is invalid.
  2. Delete the corrupt file and restart the daemon: BestEffortLongFile is best-effort by design, so it re-initializes with the default and is rewritten on the next set() (verify the lost value is recoverable from edit logs / other journal nodes first).
  3. If the value must be preserved, restore the 8-byte file from a storage-directory backup.
  4. Fix the root cause: free disk space, move storage to a healthy local disk, and stop manual edits of journal files.

Example fix

// before
BestEffortLongFile committedTxnId = new BestEffortLongFile(file, 0);
long v = committedTxnId.get(); // throws 'had invalid length: 3' on a torn file

// after - detect a torn file before open, quarantine it, fall back to default
if (file.exists() && file.length() != 0 && file.length() != 8) {
  LOG.warn("Quarantining corrupt {} ({} bytes)", file, file.length());
  Files.move(file.toPath(), file.toPath().resolveSibling(file.getName() + ".corrupt"));
}
BestEffortLongFile committedTxnId = new BestEffortLongFile(file, 0);
long v = committedTxnId.get();
Defensive patterns

Strategy: validation

Validate before calling

static void ensureValidLongFile(File f) throws IOException {
  if (f.exists() && f.length() != 0 && f.length() != 8) {
    throw new IOException(String.format(
        "Refusing to open %s: %d bytes (expected 0 or 8)", f, f.length()));
  }
}

Try / catch

try {
  return longFile.get();
} catch (IOException e) { // 'had invalid length'
  // value is best-effort by contract: quarantine and reinitialize with default
  Files.move(file.toPath(), file.toPath().resolveSibling(file.getName() + ".corrupt"));
  longFile.close();
  longFile = new BestEffortLongFile(file, defaultVal);
  return longFile.get();
}

Prevention

When it happens

Trigger: First get()/set() (lazyOpen) after a JournalNode restart when the backing file (committed-txnid) exists with 1-7 or >8 bytes: torn write on crash (the class does not fsync or use atomic rename), disk corruption, full-disk partial allocation, or manual editing with a text editor.

Common situations: JournalNode storage directory after power loss or a killed process; storage on a filesystem that lost writes; an operator echoing text into the file; disk-full events during set().

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/6d125b36fd02153a. Report an issue: GitHub.