apache/hadoop · error · IOException

Har: completeLocalOutput not allowed

Error message

Har: completeLocalOutput not allowed

What it means

completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) is the commit half of the local-staging protocol: after data is written to the temp file, FileUtil.copy calls it to move the temp into final place. Since staging into a har can never have succeeded (startLocalOutput already throws), reaching this call means custom protocol code is trying to commit into an immutable archive.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java:866

    FileUtil.copy(this, src, getLocal(getConf()), dst, false, getConf());
  }
  
  /**
   * not implemented.
   */
  @Override
  public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) 
    throws IOException {
    throw new IOException("Har: startLocalOutput not allowed");
  }
  
  /**
   * not implemented.
   */
  @Override
  public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) 
    throws IOException {
    throw new IOException("Har: completeLocalOutput not allowed");
  }
  
  /**
   * not implemented.
   */
  @Override
  public void setOwner(Path p, String username, String groupname)
    throws IOException {
    throw new IOException("Har: setowner not allowed");
  }

  @Override
  public void setTimes(Path p, long mtime, long atime) throws IOException {
    throw new IOException("Har: setTimes not allowed");
  }

  /**
   * Not implemented.

View on GitHub (pinned to 2add963021)

Solutions

1. Replace the manual staging protocol with FileUtil.copy to the underlying HDFS plus `hadoop archive`
2. If wrapping filesystems, short-circuit the protocol when the destination is read-only
3. Audit custom copy code for completeLocalOutput calls and add a scheme guard

Example fix

// before
Path tmp = dstFs.startLocalOutput(harDst, tmpLocal); // replaced by custom code
writeStaged(tmp);
dstFs.completeLocalOutput(harDst, tmpLocal); // throws: Har: completeLocalOutput not allowed

// after
FileUtil.copy(localFs, src, hdfs, hdfsDst, false, conf);
// then: hadoop archive -archiveName data.har -p <dir> <parent>

if ("har".equals(fsOutputFile.toUri().getScheme())) {
  throw new UnsupportedOperationException("completeLocalOutput is not supported on har: " + fsOutputFile);
}

When it happens

Trigger: Custom implementations of the startLocalOutput/completeLocalOutput dance invoking dstFs.completeLocalOutput(harPath, tmp) — FileUtil.copy itself never gets here because startLocalOutput throws first.

Common situations: Hand-rolled large-file upload logic copied from FileSystem/FileUtil internals; wrapper filesystems replaying the staging protocol against an inner HarFileSystem.

Related errors

• Har: startLocalOutput not allowed
• Har: create not allowed.
• Har: append not allowed.
• Har: setReplication not allowed
• Har: rename not allowed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/af4c644f7d55874d. Report an issue: GitHub.