apache/hadoop · error · PathIOException

Dest filesystem '${fs.getUri().getScheme()}' doesn't support

Error message

Dest filesystem '${fs.getUri().getScheme()}' doesn't support concat.

What it means

PathIOException thrown by Concat.java:82 when the underlying FileSystem.concat(target, srcArray) call raises UnsupportedOperationException. The shell catches it and reports which scheme failed: concat is a block-level merge that only some filesystem implementations (primarily HDFS) support; LocalFileSystem, ViewFs, and most object-store implementations do not implement it.

Source

Thrown at hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/shell/Concat.java:82

              target.path));
    }
    Path[] srcArray = new Path[srcList.size()];
    for (int i = 0; i < args.size(); i++) {
      PathData src = srcList.get(i);
      if (!src.exists || !src.stat.isFile()) {
        throw new FileNotFoundException(
            String.format("%s does not exist or is not file.", src.path));
      }
      srcArray[i] = src.path;
    }
    FileSystem fs = target.fs;
    if (testFs != null) {
      fs = testFs;
    }
    try {
      fs.concat(target.path, srcArray);
    } catch (UnsupportedOperationException exception) {
      throw new PathIOException("Dest filesystem '" + fs.getUri().getScheme()
          + "' doesn't support concat.", exception);
    }
  }

  @VisibleForTesting
  static void setTestFs(FileSystem fs) {
    testFs = fs;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

Run the command against an HDFS target: 'hdfs dfs -concat hdfs://nn/data/target hdfs://nn/data/src1 hdfs://nn/data/src2'
If you only need the byte-level result on a non-HDFS filesystem, use 'hadoop fs -getmerge' or copy+append instead
In Java code, guard with try/catch UnsupportedOperationException around fs.concat(), or check 'hdfs'.equals(fs.getUri().getScheme()) first
Verify fs.defaultFS in core-site.xml points at the intended HDFS cluster when the command is given unqualified paths

Example fix

// before
fs.concat(target, srcs);  // fs is RawLocalFileSystem -> UnsupportedOperationException

// after
if ("hdfs".equals(fs.getUri().getScheme())) {
  fs.concat(target, srcs);
} else {
  throw new UnsupportedOperationException(
      "Dest filesystem '" + fs.getUri().getScheme() + "' doesn't support concat.");
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (!"hdfs".equals(fs.getUri().getScheme())) {
  throw new UnsupportedOperationException(
      "Dest filesystem '" + fs.getUri().getScheme() + "' doesn't support concat.");
}

Type guard

static boolean supportsConcat(FileSystem fs) {
  return "hdfs".equals(fs.getUri().getScheme());
}

Try / catch

try {
  fs.concat(target, srcArray);
} catch (UnsupportedOperationException e) {
  // fall back to a content-level merge (getmerge / copy+append)
}

Prevention

Treat concat as an HDFS-only optimization; gate it on the target filesystem scheme
Set fs.defaultFS explicitly so unqualified paths cannot silently resolve to file://
In tests, use a MiniDFSCluster-based FileSystem, not RawLocalFileSystem, when exercising concat

When it happens

Trigger: 'hadoop fs -concat /tmp/t /tmp/a /tmp/b' where /tmp resolves to the local filesystem (file://); the command run against viewfs:, s3a://, or another non-HDFS default FS; unit tests where Concat.setTestFs() injected a RawLocalFileSystem. Note the filesystem used is the TARGET's fs, so the scheme in the message is the target's scheme.

Common situations: Running shell commands on a gateway node whose fs.defaultFS is file:/// or a non-HDFS URI; scripts written for HDFS reused against local paths during testing; code calling FileSystem.get(conf).concat() on a FileSystem implementation that does not override it (the base class throws UnsupportedOperationException by design).

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/3eaea51e9ee86357. Report an issue: GitHub.