apache/hadoop · error · IOException

Couldn't run retriable-command: {}

Error message

Couldn't run retriable-command: {}

What it means

RetriableCommand.execute() loops the command through a RetryPolicy; when the policy stops advising RETRY (retries exhausted or the error judged non-retryable), it gives up and throws this IOException carrying the command description and the latest underlying exception. RetriableFileCopyCommand and RetriableDirectoryCreateCommand both surface failures through it, usually further wrapped by CopyMapper.

Source

Thrown at hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java:101

    Exception latestException;
    int counter = 0;
    while (true) {
      try {
        return doExecute(arguments);
      } catch(Exception exception) {
        LOG.error("Failure in Retriable command: " + description, exception);
        latestException = exception;
      }
      counter++;
      RetryAction action = retryPolicy.shouldRetry(latestException, counter, 0, true);
      if (action.action == RetryPolicy.RetryAction.RetryDecision.RETRY) {
        ThreadUtil.sleepAtLeastIgnoreInterrupts(action.delayMillis);
      } else {
        break;
      }
    }

    throw new IOException("Couldn't run retriable-command: " + description,
                          latestException);
  }

  /**
   * Fluent-interface to change the RetryHandler.
   * @param retryHandler The new RetryHandler instance to be used.
   * @return Self.
   */
  public RetriableCommand setRetryPolicy(RetryPolicy retryHandler) {
    this.retryPolicy = retryHandler;
    return this;
  }
}

View on GitHub (pinned to 2add963021)

Solutions

1. Inspect latestException (the cause) - it names the real, final failure that exhausted the retries
2. Fix the persistent root cause (permissions, missing paths, space/quota) and re-run with -update
3. For genuinely flaky environments, widen the budget via setRetryPolicy (e.g. exponential backoff) instead of manual re-runs
4. If the cause is non-retryable by design (FileNotFoundException), correct the input listing instead of retrying

Example fix

// before: default retry budget too small for a flaky cluster
new RetriableDirectoryCreateCommand("mkdir")
    .execute(target, context, sourceStatus, sourceFS);

// after: widen the retry window
new RetriableDirectoryCreateCommand("mkdir")
    .setRetryPolicy(RetryPolicies.exponentialBackoffRetry(
        20, 500, TimeUnit.MILLISECONDS))
    .execute(target, context, sourceStatus, sourceFS);

catch (IOException e) {  // 'Couldn't run retriable-command: <description>'
  Throwable latest = e.getCause();  // the actual final failure after retries
  LOG.error("retriable command gave up, cause:", latest);
}

When it happens

Trigger: A persistent error (permission denied, file not found, no space) that never becomes retryable; a cluster outage lasting longer than the retry budget; a custom RetryPolicy installed via setRetryPolicy() that fails fast.

Common situations: Target permissions fixed too late; source files deleted mid-job; DataNode/NameNode incidents exceeding the retry window; users tightening the retry policy to speed jobs up.

Related errors

• File copy failed: {sourcePath} --> {target}
• key + ": No such file or directory."
• %s: Stream is closed!
• Stream is closed!
• Cannot seek to a negative offset

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/b2b8a7e6ff4aa7fe. Report an issue: GitHub.