What causes "Parent directory doesn't exist: " + parent" in apache/hadoop?

Verify framing integrity: every readRaw() must start exactly at a record boundary — audit any manual in.skip()/read() between records.

apache/hadoop · error · FileNotFoundException

Parent directory doesn't exist: " + parent

Error message

Parent directory doesn't exist: " + parent

What it means

TypedBytesInput.readRaw() is the low-level variant that returns the raw typed-bytes encoding (including the type byte) instead of a Java object. It accepts the same code set as read() — core types, LIST/MAP/VECTOR raw forms, MARKER returning null, and 50-200 application codes via readRawBytes(code) — and throws RuntimeException 'unknown type' for any other leading byte. As with read(), the exception signals stream misalignment or a non-typed-bytes source rather than a fixable type problem.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:236

   * @throws IOException if an I/O error occurs
   */
  @Override
  public FSDataOutputStream createNonRecursive(Path f,
      FsPermission permission, boolean overwrite,
      int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    Path absolutePath = makeAbsolute(f);
    Path parent = absolutePath.getParent();

    if (parent != null && !parent.isRoot()) {
      try {
        FileStatus parentStatus = getFileStatus(parent);
        if (!parentStatus.isDirectory()) {
          throw new FileAlreadyExistsException(
              parent + " is a file");
        }
      } catch (FileNotFoundException e) {
        throw new FileNotFoundException(
            "Parent directory doesn't exist: " + parent);
      }
    }

    return create(f, permission, overwrite, bufferSize,
        replication, blockSize, progress);
  }

  /**
   * Create a file non-recursively with CreateFlag set.
   *
   * @param f the file name to create
   * @param permission the permission to set
   * @param flags creation flags
   * @param bufferSize the buffer size
   * @param replication the replication factor
   * @param blockSize the block size
   * @param progress for reporting progress

View on GitHub (pinned to 2add963021)

Solutions

Verify framing integrity: every readRaw() must start exactly at a record boundary — audit any manual in.skip()/read() between records.
Ensure the producer writes with TypedBytesOutput (or a spec-compliant writer) and that both sides use the same typed-bytes dialect.
For custom protocols keep codes within 50-200 so readRaw() returns them as raw bytes instead of throwing.
Debug by hex-dumping bytes around the failure offset to find the first desynchronized record and fix the writer side.

Defensive patterns

Strategy: try-catch

Try / catch

try {
  Buffer b = tIn.readRaw();
} catch (RuntimeException e) {
  if ("unknown type".equals(e.getMessage())) {
    // resync logic: drop to next known boundary or fail the split
    throw new IOException("typed-bytes raw stream out of sync", e);
  }
  throw e;
}

Prevention

Read whole records only; never partial-skip between readRaw calls.
Validate the producer uses TypedBytesOutput with the same dialect.
Hex-dump at the failure offset to find the first bad record.

When it happens

Trigger: Calling readRaw() on desynchronized input: after skipping only part of a previous record, reading a stream produced by a text writer, or consuming bytes produced by readRawBytes for a custom code your reader build does not understand (codes >200 other than 255).

Common situations: Custom InputFormat/RecordReader built on TypedBytesInput.readRaw() for pipes; feeding files from a different serialization (Hadoop Writable files, raw text) into it; partial reads where the caller consumed the type byte separately and then calls readRaw again mid-record.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/ab6fa7991b2ca98c. Report an issue: GitHub.