apache/hadoop · error · IOException

Har: completeLocalOutput not allowed

Error message

Har: completeLocalOutput not allowed

What it means

completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) is the commit half of the local-staging protocol: after data is written to the temp file, FileUtil.copy calls it to move the temp into final place. Since staging into a har can never have succeeded (startLocalOutput already throws), reaching this call means custom protocol code is trying to commit into an immutable archive.

Solutions

1. Replace the manual staging protocol with FileUtil.copy to the underlying HDFS plus `hadoop archive`
2. If wrapping filesystems, short-circuit the protocol when the destination is read-only
3. Audit custom copy code for completeLocalOutput calls and add a scheme guard

Example fix

// before
Path tmp = dstFs.startLocalOutput(harDst, tmpLocal); // replaced by custom code
writeStaged(tmp);
dstFs.completeLocalOutput(harDst, tmpLocal); // throws: Har: completeLocalOutput not allowed

// after
FileUtil.copy(localFs, src, hdfs, hdfsDst, false, conf);
// then: hadoop archive -archiveName data.har -p <dir> <parent>

if ("har".equals(fsOutputFile.toUri().getScheme())) {
  throw new UnsupportedOperationException("completeLocalOutput is not supported on har: " + fsOutputFile);
}

When it happens

Trigger: Custom implementations of the startLocalOutput/completeLocalOutput dance invoking dstFs.completeLocalOutput(harPath, tmp) — FileUtil.copy itself never gets here because startLocalOutput throws first.

Common situations: Hand-rolled large-file upload logic copied from FileSystem/FileUtil internals; wrapper filesystems replaying the staging protocol against an inner HarFileSystem.

Related errors

• Har: startLocalOutput not allowed
• Har: append not allowed.
• Har: copyfromlocalfile not allowed
• Har: create not allowed.
• Har: delete not allowed

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/af4c644f7d55874d. Report an issue: GitHub.

Appendix: source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/HarFileSystem.java:866

    FileUtil.copy(this, src, getLocal(getConf()), dst, false, getConf());
  }
  
  /**
   * not implemented.
   */
  @Override
  public Path startLocalOutput(Path fsOutputFile, Path tmpLocalFile) 
    throws IOException {
    throw new IOException("Har: startLocalOutput not allowed");
  }
  
  /**
   * not implemented.
   */
  @Override
  public void completeLocalOutput(Path fsOutputFile, Path tmpLocalFile) 
    throws IOException {
    throw new IOException("Har: completeLocalOutput not allowed");
  }
  
  /**
   * not implemented.
   */
  @Override
  public void setOwner(Path p, String username, String groupname)
    throws IOException {
    throw new IOException("Har: setowner not allowed");
  }

  @Override
  public void setTimes(Path p, long mtime, long atime) throws IOException {
    throw new IOException("Har: setTimes not allowed");
  }

  /**
   * Not implemented.

View on GitHub (pinned to 2add963021)