What causes "parent + " is a file"" in apache/hadoop?

Make both ends of the pipe agree on the format: pass -io typedbytes to every streaming stage (mapper, combiner, reducer) so both write and read typed bytes.

apache/hadoop · error · FileAlreadyExistsException

parent + " is a file"

Error message

parent + " is a file"

What it means

TypedBytesInput.read() interprets the next byte of the stream as a typed-bytes type code and dispatches to the matching reader. Codes 0-10 map to the core types, MARKER (255) returns null, 50-200 are treated as application-specific byte payloads, and anything else hits the else branch throwing RuntimeException 'unknown type'. Hitting it means the stream is not aligned on a typed-bytes record — you are reading garbage, not that the type is merely unsupported.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:232

   * @param replication the replication factor
   * @param blockSize the block size
   * @param progress for reporting progress
   * @return an output stream to write to
   * @throws IOException if an I/O error occurs
   */
  @Override
  public FSDataOutputStream createNonRecursive(Path f,
      FsPermission permission, boolean overwrite,
      int bufferSize, short replication, long blockSize,
      Progressable progress) throws IOException {
    Path absolutePath = makeAbsolute(f);
    Path parent = absolutePath.getParent();

    if (parent != null && !parent.isRoot()) {
      try {
        FileStatus parentStatus = getFileStatus(parent);
        if (!parentStatus.isDirectory()) {
          throw new FileAlreadyExistsException(
              parent + " is a file");
        }
      } catch (FileNotFoundException e) {
        throw new FileNotFoundException(
            "Parent directory doesn't exist: " + parent);
      }
    }

    return create(f, permission, overwrite, bufferSize,
        replication, blockSize, progress);
  }

  /**
   * Create a file non-recursively with CreateFlag set.
   *
   * @param f the file name to create
   * @param permission the permission to set
   * @param flags creation flags

View on GitHub (pinned to 2add963021)

Solutions

Make both ends of the pipe agree on the format: pass -io typedbytes to every streaming stage (mapper, combiner, reducer) so both write and read typed bytes.
Confirm the data source is genuinely typed-bytes encoded before wrapping it in TypedBytesInput; if it is raw text, use Text/LineReader-based readers instead.
If you control the protocol, confine custom codes to the 50-200 application-specific range, which read() accepts as raw Buffer payloads.
Wrap per-record reads in a small try/catch that logs the offending byte (mark/reset the stream) to identify where desynchronization starts.

Example fix

# before (bash streaming: only one side typed)
hadoop jar streaming.jar -mapper my_mapper.py -io typedbytes -reducer 'wc -l' ...

# after
hadoop jar streaming.jar -mapper my_mapper.py -io typedbytes \
  -reducer my_reducer.py -io typedbytes ...

Defensive patterns

Strategy: try-catch

Validate before calling

// peek the type code before committing to read()
in.mark(1);
int code = in.readUnsignedByte();
in.reset();
boolean known = (0 <= code && code <= 10) || code == 255 || (50 <= code && code <= 200);
if (!known) throw new IOException("not a typed-bytes stream at offset ...");

Try / catch

try {
  Object o = tIn.read();
} catch (RuntimeException e) {
  if (e.getMessage().equals("unknown type")) {
    throw new IOException("typed-bytes stream desynchronized", e); // treat as corruption, not retryable
  }
  throw e;
}

Prevention

Set -io typedbytes on every streaming stage or none.
Never interleave raw reads with TypedBytesInput on the same stream.
Keep custom type codes inside 50-200.

When it happens

Trigger: Calling TypedBytesInput.read() (directly or via TypedBytesRecordReader/typed-bytes streaming pipes) on a stream that is out of sync: after partially consuming a record, after mixing raw text and typed-bytes modes, or when the upstream writer did not emit typed bytes at all (e.g. -io typedbytes missing on one side).

Common situations: Streaming jobs where mapper output is typed bytes but the reducer is a plain script (or vice versa) because -io typedbytes was set on only one command; deserializing a file that was never typed-bytes encoded; a Python/Ruby streaming script printing raw strings into a typed-bytes channel; version drift where a peer emits a code outside 0-10/50-200/255.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/e965cd143162db6b. Report an issue: GitHub.