{"record":{"id":"b2b8a7e6ff4aa7fe","repo":"apache/hadoop","slug":"couldn-t-run-retriable-command","errorCode":null,"errorMessage":"Couldn't run retriable-command: {}","messagePattern":"Couldn't run retriable-command: (.+?)","errorType":"exception","errorClass":"IOException","httpStatus":null,"severity":"error","filePath":"hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java","lineNumber":101,"sourceCode":"    Exception latestException;\n    int counter = 0;\n    while (true) {\n      try {\n        return doExecute(arguments);\n      } catch(Exception exception) {\n        LOG.error(\"Failure in Retriable command: \" + description, exception);\n        latestException = exception;\n      }\n      counter++;\n      RetryAction action = retryPolicy.shouldRetry(latestException, counter, 0, true);\n      if (action.action == RetryPolicy.RetryAction.RetryDecision.RETRY) {\n        ThreadUtil.sleepAtLeastIgnoreInterrupts(action.delayMillis);\n      } else {\n        break;\n      }\n    }\n\n    throw new IOException(\"Couldn't run retriable-command: \" + description,\n                          latestException);\n  }\n\n  /**\n   * Fluent-interface to change the RetryHandler.\n   * @param retryHandler The new RetryHandler instance to be used.\n   * @return Self.\n   */\n  public RetriableCommand setRetryPolicy(RetryPolicy retryHandler) {\n    this.retryPolicy = retryHandler;\n    return this;\n  }\n}\n","sourceCodeStart":83,"sourceCodeEnd":115,"githubUrl":"https://github.com/apache/hadoop/blob/2add9630210752f88ceb1bb74eb65e37bf41da8e/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/RetriableCommand.java#L83-L115","documentation":"RetriableCommand.execute() loops the command through a RetryPolicy; when the policy stops advising RETRY (retries exhausted or the error judged non-retryable), it gives up and throws this IOException carrying the command description and the latest underlying exception. RetriableFileCopyCommand and RetriableDirectoryCreateCommand both surface failures through it, usually further wrapped by CopyMapper.","triggerScenarios":"A persistent error (permission denied, file not found, no space) that never becomes retryable; a cluster outage lasting longer than the retry budget; a custom RetryPolicy installed via setRetryPolicy() that fails fast.","commonSituations":"Target permissions fixed too late; source files deleted mid-job; DataNode/NameNode incidents exceeding the retry window; users tightening the retry policy to speed jobs up.","solutions":["Inspect latestException (the cause) - it names the real, final failure that exhausted the retries","Fix the persistent root cause (permissions, missing paths, space/quota) and re-run with -update","For genuinely flaky environments, widen the budget via setRetryPolicy (e.g. exponential backoff) instead of manual re-runs","If the cause is non-retryable by design (FileNotFoundException), correct the input listing instead of retrying"],"exampleFix":"// before: default retry budget too small for a flaky cluster\nnew RetriableDirectoryCreateCommand(\"mkdir\")\n    .execute(target, context, sourceStatus, sourceFS);\n\n// after: widen the retry window\nnew RetriableDirectoryCreateCommand(\"mkdir\")\n    .setRetryPolicy(RetryPolicies.exponentialBackoffRetry(\n        20, 500, TimeUnit.MILLISECONDS))\n    .execute(target, context, sourceStatus, sourceFS);","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"catch (IOException e) {  // 'Couldn't run retriable-command: <description>'\n  Throwable latest = e.getCause();  // the actual final failure after retries\n  LOG.error(\"retriable command gave up, cause:\", latest);\n}","preventionTips":["Fix persistent causes (permissions, missing paths, space) before relying on retries","Size the RetryPolicy to your cluster's worst realistic outage window","Re-run with -update so completed work is not repeated","Distinguish non-retryable causes and fix the listing/input instead"],"tags":["distcp","hadoop","retry","io"],"backgroundTag":"retry-exhausted","analyzedSha":"2add9630210752f88ceb1bb74eb65e37bf41da8e","analyzedAt":"2026-08-22T19:55:07.957Z","schemaVersion":2},"datasetVersion":"2026-08-22T20:17:22.307Z"}