apache/hadoop · error · FileAlreadyExistsException

f + " already exists"

Error message

f + " already exists"

What it means

The mapreduce-API copy of StreamXmlRecordReader (org.apache.hadoop.streaming.mapreduce) mirrors the legacy one: its init reads CONF_NS+'begin' and CONF_NS+'end' through checkJobGet, and checkJobGet throws IOException 'JobConf: missing required property' when conf_.get(prop) is null. Both marker properties are mandatory for XML-style record extraction.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:195

    };
  }

  @Override
  public FSDataOutputStream create(Path f, FsPermission permission,
      boolean overwrite, int bufferSize, short replication,
      long blockSize, Progressable progress) throws IOException {
    Path absolutePath = makeAbsolute(f);
    String key = pathToKey(absolutePath);

    if (overwrite) {
      if (store.isDirectory(key)) {
        throw new FileAlreadyExistsException(
            f + " is a directory");
      }
    } else {
      try {
        getFileStatus(f);
        throw new FileAlreadyExistsException(
            f + " already exists");
      } catch (FileNotFoundException e) {
        // nothing, ok
      }
    }

    return new FSDataOutputStream(
        store.createFile(key, getConf()), statistics);
  }

  /**
   * Create a file non-recursively. The parent directory must
   * already exist.
   *
   * @param f the file name to create
   * @param permission the permission to set
   * @param overwrite if true, overwrite the existing file
   * @param bufferSize the buffer size

View on GitHub (pinned to 2add963021)

Solutions

1. Add both -jobconf stream.recordreader.begin=<beginRegex> and -jobconf stream.recordreader.end=<endRegex> to the streaming command.
2. Confirm the properties survive to the task side: dump the effective config or read them back via job.getConfiguration().get(...) before submit.
3. Optional tuning keys that also live under stream.recordreader.: maxrec (default 50000) and lookahead (default 2*maxrec) — check them for typos too.
4. If you only need line records, use the default reader instead of StreamXmlRecordReader.

Example fix

// before
job.getConfiguration().set("stream.recordreader.class",
    "org.apache.hadoop.streaming.mapreduce.StreamXmlRecordReader");

// after
job.getConfiguration().set("stream.recordreader.class",
    "org.apache.hadoop.streaming.mapreduce.StreamXmlRecordReader");
job.getConfiguration().set("stream.recordreader.begin", "<record>");
job.getConfiguration().set("stream.recordreader.end", "</record>");

Configuration conf = job.getConfiguration();
if (conf.get("stream.recordreader.class", "").contains("StreamXmlRecordReader")) {
  Preconditions.checkState(conf.get("stream.recordreader.begin") != null,
      "stream.recordreader.begin must be set");
  Preconditions.checkState(conf.get("stream.recordreader.end") != null,
      "stream.recordreader.end must be set");
}

When it happens

Trigger: Using -inputreader org.apache.hadoop.streaming.mapreduce.StreamXmlRecordReader without -jobconf stream.recordreader.begin / stream.recordreader.end, or setting them in a config file that is not actually loaded into the task's Configuration (conf_).

Common situations: Same family as the mapred variant: forgotten marker properties after switching from LineRecordReader; properties defined only on the client-side JobConf but overwritten by a wrapper (Hive/Pig streaming) that rebuilds the configuration; typo'd property keys.

Related errors

• No more entry in " + f
• f + " is a directory"
• Expression is null
• Expression is null
• Path must be absolute: " + path

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fb3b55b225ce1452. Report an issue: GitHub.