apache/hadoop · error · IOException

The destination cluster must be specified.

Error message

The destination cluster must be specified.

What it means

FedBalance.BalanceJob.Builder.build() requires a fully-qualified destination URI: the authority component (nameservice or namenode host:port) identifies the destination cluster, which is mandatory because fedbalance copies across clusters. If new Path(inputDst).toUri().getAuthority() is null - a relative path or one without an authority - the IOException is thrown before any job object is built.

Source

Thrown at hadoop-tools/hadoop-federation-balance/src/main/java/org/apache/hadoop/tools/fedbalance/FedBalance.java:158

    /**
     * Specify the threshold of diff entries.
     * @param value the threshold of a fast distcp.
     */
    public Builder setDiffThreshold(int value) {
      this.diffThreshold = value;
      return this;
    }

    /**
     * Build the balance job.
     */
    public BalanceJob build() throws IOException {
      // Construct job context.
      FedBalanceContext context;
      Path dst = new Path(inputDst);
      if (dst.toUri().getAuthority() == null) {
        throw new IOException("The destination cluster must be specified.");
      }
      Path src = new Path(inputSrc);
      if (src.toUri().getAuthority() == null) {
        throw new IOException("The source cluster must be specified.");
      }
      context = new FedBalanceContext.Builder(src, dst, NO_MOUNT, getConf())
          .setForceCloseOpenFiles(forceCloseOpen).setUseMountReadOnly(false)
          .setMapNum(map).setBandwidthLimit(bandwidth).setTrash(trashOpt)
          .setDiffThreshold(diffThreshold).build();

      LOG.info(context.toString());
      // Construct the balance job.
      BalanceJob.Builder<BalanceProcedure> builder = new BalanceJob.Builder<>();
      DistCpProcedure dcp =
          new DistCpProcedure(DISTCP_PROCEDURE, null, delayDuration, context);
      builder.nextProcedure(dcp);
      TrashProcedure tp =
          new TrashProcedure(TRASH_PROCEDURE, null, delayDuration, context);

View on GitHub (pinned to 2add963021)

Solutions

1. Set -dst to a fully-qualified URI: hdfs://<nameservice-or-namenode>:<port>/path.
2. Confirm the nameservice you name is defined in the client's hdfs-site.xml (dfs.nameservices / dfs.ha.namenodes).
3. Apply the same fully-qualified form to -src, which is checked next and fails with its own message.

Example fix

# before
hdfs fedbalance -src hdfs://ns1/data -dst /balance/data

# after
hdfs fedbalance -src hdfs://ns1/data -dst hdfs://ns2/balance/data

String dstStr = "hdfs://ns2/balance/data";
URI u = new Path(dstStr).toUri();
if (u.getAuthority() == null) {
  throw new IllegalArgumentException("fedbalance -dst must include the cluster authority, e.g. hdfs://ns2/path");
}

When it happens

Trigger: Passing -dst /some/path (no scheme/authority), or a URI form whose authority is empty, to the fedbalance CLI. dst.toUri().getAuthority() returns null and build() fails.

Common situations: Copy-pasting a plain path that works with 'hdfs dfs -cp'; forgetting the nameservice in router-federation deployments; assuming fs.defaultFS will be applied (fedbalance deliberately does not infer it).

Related errors

• The source cluster must be specified.
• The destination cluster must be specified.
• {src} should be a directory.
• {dst} already exists.
• {src} shouldn't enable snapshot.

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/254e3d62e2601bcb. Report an issue: GitHub.