apache/hadoop · error · InvalidInputException

The source path '{path}' starts with /.reserved/raw but the

Error message

The source path '{path}' starts with /.reserved/raw but the target path '{targetPath}' does not. Either all or none of the paths must have this prefix.

What it means

DistCp requires the /.reserved/raw prefix to be used consistently on source and target. A source under /.reserved/raw exposes the raw block-level view including raw* xattrs (used for encryption-zone key material), while a normal target cannot store them; copying in that direction would silently drop or corrupt raw xattrs, so listing aborts.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java:190

    if (context.shouldAtomicCommit() && targetExists) {
      throw new InvalidInputException("Target path for atomic-commit already exists: " +
        targetPath + ". Cannot atomic-commit to pre-existing target-path.");
    }

    for (Path path: context.getSourcePaths()) {
      FileSystem fs = path.getFileSystem(getConf());
      if (!fs.exists(path)) {
        throw new InvalidInputException(path + " doesn't exist");
      }
      if (Path.getPathWithoutSchemeAndAuthority(path).toString().
          startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME)) {
        if (!targetIsReservedRaw) {
          final String msg = "The source path '" + path + "' starts with " +
              HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the target path '" +
              targetPath + "' does not. Either all or none of the paths must " +
              "have this prefix.";
          throw new InvalidInputException(msg);
        }
      } else if (targetIsReservedRaw) {
        final String msg = "The target path '" + targetPath + "' starts with " +
                HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the source path '" +
                path + "' does not. Either all or none of the paths must " +
                "have this prefix.";
        throw new InvalidInputException(msg);
      }
    }

    if (targetIsReservedRaw) {
      context.setPreserveRawXattrs(true);
      getConf().setBoolean(DistCpConstants.CONF_LABEL_PRESERVE_RAWXATTRS, true);
    }

    /* This is requires to allow map tasks to access each of the source
       clusters. This would retrieve the delegation token for each unique
       file system and add them to job's private credential store

View on GitHub (pinned to 2add963021)

Solutions

Add the same prefix to the target: hadoop distcp hdfs://nn/.reserved/raw/ez/src hdfs://nn2/.reserved/raw/backup.
Or drop /.reserved/raw from the source if you intentionally want the normal (encrypted/transparent) view copied.
Verify every source path when using globs or -f listings - all of them must match the target's raw-ness.

Example fix

# before: raw source, normal target
hadoop distcp hdfs://nn/.reserved/raw/zone/src hdfs://nn2/backup

# after: both sides raw
hadoop distcp hdfs://nn/.reserved/raw/zone/src hdfs://nn2/.reserved/raw/backup

Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight: raw prefix must match between every source and the target
String RAW = "/.reserved/raw";
String tp = Path.getPathWithoutSchemeAndAuthority(targetPath).toString();
boolean targetRaw = tp.startsWith(RAW);
for (Path p : context.getSourcePaths()) {
  boolean srcRaw = Path.getPathWithoutSchemeAndAuthority(p).toString().startsWith(RAW);
  if (srcRaw != targetRaw) {
    throw new InvalidInputException(
        "Raw prefix mismatch between " + p + " and " + targetPath);
  }
}

Try / catch

try {
  copyListing.doBuildListing(listingFile, context);
} catch (InvalidInputException e) {
  if (e.getMessage().contains("Either all or none of the paths must have this prefix")) {
    // add or strip /.reserved/raw so source and target match, then re-run
    alignRawPrefixesAndRerun();
  } else {
    throw e;
  }
}

Prevention

When copying encryption zones at block level, put /.reserved/raw on BOTH source and target.
Check every path in -f listings and globs, not just the first source.
Keep raw-path policies in one script constant so both sides are updated together.

When it happens

Trigger: hadoop distcp hdfs://nn/.reserved/raw/ez/src hdfs://nn2/backup - the source carries the raw prefix, the target does not. Any -sync or plain copy mixing one raw-prefixed source with a non-raw target.

Common situations: backing up encryption zones: operators quote the raw source path but a normal destination; scripts templating the target without the raw prefix; partially migrated runbooks where only one side was updated.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/1193e4764299921e. Report an issue: GitHub.