apache/hadoop · error · UnsupportedOperationException

Append is not supported by BaiduBosFileSystem

Error message

Append is not supported by BaiduBosFileSystem

What it means

StreamUtil's class-location logic resolves where a class was loaded from (jar path or class directory) so streaming can ship that code to task nodes. After normalizing the resource URL it strips the trailing '/my/package/Class.class' by searching for the relative class path; if lastIndexOf(relPath) returns -1 it throws IllegalArgumentException 'invalid codePath'. This means the classloader produced a URL that does not end with the package-relative path of the class, so the jar/root cannot be derived.

Source

Thrown at hadoop-cloud-storage-project/hadoop-bos/src/main/java/org/apache/hadoop/fs/bos/BaiduBosFileSystem.java:151

    if (path.isAbsolute()) {
      return path;
    }
    return new Path(workingDir, path);
  }

  /**
   * This optional operation is not yet supported.
   *
   * @param f the file to append to
   * @param bufferSize the buffer size
   * @param progress for reporting progress
   * @return never returns
   * @throws IOException always thrown
   */
  @Override
  public FSDataOutputStream append(Path f, int bufferSize,
      Progressable progress) throws IOException {
    throw new UnsupportedOperationException(
        "Append is not supported by BaiduBosFileSystem");
  }

  @Override
  public RemoteIterator<LocatedFileStatus> listLocatedStatus(
      final Path f, final PathFilter filter) throws IOException {
    return new RemoteIterator<LocatedFileStatus>() {
      private final FileStatus[] stats = listStatus(f, filter);
      private int i = 0;

      public boolean hasNext() {
        return i < stats.length;
      }

      public LocatedFileStatus next() throws IOException {
        if (!hasNext()) {
          throw new NoSuchElementException(
              "No more entry in " + f);

View on GitHub (pinned to 2add963021)

Solutions

1. Avoid the auto-detection path: package your mapper in a jar and register it explicitly with -libjars / job.setJar(...), or set mapreduce.job.jar, so the streaming code path shipping is skipped.
2. If you control the launcher, ensure the class is loaded from a plain file: or jar: URL whose path literally ends with <package-path>/<Class>.class (no custom URL schemes).
3. Check for runtime-generated or proxied classes being passed as the command class and pass the real class name instead.
4. Upgrade/verify you are not hitting a classloader rewrite from an agent (e.g. coverage or APM agents prepending transformed class locations).

Example fix

# before: rely on auto-detection of class location
hadoop jar hadoop-streaming.jar -mapper com.myco.MyMapper -reducer ...

# after: ship the jar explicitly, bypassing class-location detection
hadoop jar hadoop-streaming.jar -libjars my-mapper.jar \
  -mapper com.myco.MyMapper -input ... -output ...

java.net.URL u = loader.getResource(className.replace('.', '/') + ".class");
if (u == null || !u.toString().contains(className.replace('.', '/') + ".class")) {
  // auto-detection will fail; fall back to explicit jar shipping
  job.setJar("my-mapper.jar");
}

try {
  String jar = StreamUtil.stripClass(mapperClassName);
} catch (IllegalArgumentException e) {
  // classloader URL not strip-able: ship jar explicitly instead of guessing
  jobConf.setJarByClass(MapperClass.class);
}

When it happens

Trigger: Calling StreamUtil.stripClass(className) (used when auto-shipping -mapper/-reducer Java classes) when the class resource URL has been rewritten by a non-standard classloader — e.g. agents, proxies (Hibernate/CGLIB style), Spring boot executable-jar loaders, or URLs with query/fragment suffixes so 'package/Class.class' no longer literally matches.

Common situations: Running streaming with a Java mapper/reducer inside an application container or with -libjars plus a custom ClassLoader; classes generated at runtime; shaded/renamed packages where the .class resource path differs from the Class.getName(); JPMS/module classpath URLs.

Related errors

• Path must be absolute: " + path
• No more entry in " + f
• f + " is a directory"
• f + " already exists"
• parent + " is a file"

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3d9e03f9588c4d95. Report an issue: GitHub.