apache/hadoop · error · InvalidInputException

The target path '{targetPath}' starts with /.reserved/raw bu

Error message

The target path '{targetPath}' starts with /.reserved/raw but the source path '{path}' does not. Either all or none of the paths must have this prefix.

What it means

The inverse raw-path mismatch: the target starts with /.reserved/raw but this source path does not. DistCp requires all-or-none use of the raw prefix; copying a normal source into a raw destination would produce a raw view without the corresponding raw xattrs, so listing aborts.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/SimpleCopyListing.java:197

      FileSystem fs = path.getFileSystem(getConf());
      if (!fs.exists(path)) {
        throw new InvalidInputException(path + " doesn't exist");
      }
      if (Path.getPathWithoutSchemeAndAuthority(path).toString().
          startsWith(HDFS_RESERVED_RAW_DIRECTORY_NAME)) {
        if (!targetIsReservedRaw) {
          final String msg = "The source path '" + path + "' starts with " +
              HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the target path '" +
              targetPath + "' does not. Either all or none of the paths must " +
              "have this prefix.";
          throw new InvalidInputException(msg);
        }
      } else if (targetIsReservedRaw) {
        final String msg = "The target path '" + targetPath + "' starts with " +
                HDFS_RESERVED_RAW_DIRECTORY_NAME + " but the source path '" +
                path + "' does not. Either all or none of the paths must " +
                "have this prefix.";
        throw new InvalidInputException(msg);
      }
    }

    if (targetIsReservedRaw) {
      context.setPreserveRawXattrs(true);
      getConf().setBoolean(DistCpConstants.CONF_LABEL_PRESERVE_RAWXATTRS, true);
    }

    /* This is requires to allow map tasks to access each of the source
       clusters. This would retrieve the delegation token for each unique
       file system and add them to job's private credential store
     */
    Credentials credentials = getCredentials();
    if (credentials != null) {
      Path[] inputPaths = context.getSourcePaths()
          .toArray(new Path[1]);
      TokenCache.obtainTokensForNamenodes(credentials, inputPaths, getConf());
    }

View on GitHub (pinned to 2add963021)

Solutions

Add /.reserved/raw to every source: hdfs://nn/.reserved/raw/ez/src hdfs://nn2/.reserved/raw/backup.
Or remove the prefix from the target if raw semantics are not intended.
Grep the -f listing file for inconsistent prefixes: grep -c '^/.reserved/raw\|hdfs://[^/]*/.reserved/raw' list.txt vs total lines.

Example fix

# before: normal source, raw target
hadoop distcp hdfs://nn/zone/src hdfs://nn2/.reserved/raw/backup

# after: both sides raw
hadoop distcp hdfs://nn/.reserved/raw/zone/src hdfs://nn2/.reserved/raw/backup

Defensive patterns

Strategy: validation

Validate before calling

// Pre-flight: same check as source-side; catch a raw target with normal sources
String RAW = "/.reserved/raw";
boolean targetRaw = Path.getPathWithoutSchemeAndAuthority(targetPath).toString().startsWith(RAW);
for (Path p : context.getSourcePaths()) {
  boolean srcRaw = Path.getPathWithoutSchemeAndAuthority(p).toString().startsWith(RAW);
  if (targetRaw && !srcRaw) {
    throw new InvalidInputException(
        "Target is raw but source " + p + " is not");
  }
}

Try / catch

try {
  copyListing.doBuildListing(listingFile, context);
} catch (InvalidInputException e) {
  if (e.getMessage().contains("Either all or none of the paths must have this prefix")) {
    // either strip /.reserved/raw from the target or add it to all sources
    alignRawPrefixesAndRerun();
  } else {
    throw e;
  }
}

Prevention

Template restore jobs so the raw prefix is applied to all sources and the target consistently.
Audit -f listings for mixed prefixes: count lines matching /.reserved/raw vs total.

When it happens

Trigger: hadoop distcp hdfs://nn/ez/src hdfs://nn2/.reserved/raw/backup; restore jobs where the destination was templated with the raw prefix but one source (of several, or one line in the -f file) lacks it.

Common situations: disaster-recovery restores into a raw-prefixed backup area; -f listings mixing raw and normal paths; incremental scripts where the target changed to raw but sources were not updated.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/37cb0f768113a9dc. Report an issue: GitHub.