apache/hadoop · error · AbfsRestOperationException

PathNotFound

PathNotFound

Error message

Path had to be recovered from atomic rename operation.

What it means

Thrown during rename-atomicity recovery on flat-namespace (non-HNS) Azure Storage accounts. After an interrupted rename, AbfsBlobClient re-checks the source path with GetPathStatus; only HTTP 404 or HTTP 409 proves the source disappeared or changed. If that status call fails with any other error, the client cannot verify the rename and surfaces PATH_NOT_FOUND with the recovery message.

Source

Thrown at hadoop-tools/hadoop-azure/src/main/java/org/apache/hadoop/fs/azurebfs/services/AbfsBlobClient.java:1877

      renameAtomicity.redo();
      renameSrcHasChanged = false;
    } catch (AbfsRestOperationException ex) {
      /*
       * At this point, the source marked by the renamePending json file, might have
       * already got renamed by some parallel thread, or at this point, the path
       * would have got modified which would result in eTag change, which would lead
       * to a HTTP_CONFLICT. In this case, no more operation needs to be taken, and
       * the calling getPathStatus can return this source path as result.
       */
      if (ex.getStatusCode() == HTTP_NOT_FOUND
          || ex.getStatusCode() == HTTP_CONFLICT) {
        renameSrcHasChanged = true;
      } else {
        throw ex;
      }
    }
    if (!renameSrcHasChanged) {
      throw new AbfsRestOperationException(
          AzureServiceErrorCode.PATH_NOT_FOUND.getStatusCode(),
          AzureServiceErrorCode.PATH_NOT_FOUND.getErrorCode(),
          ATOMIC_DIR_RENAME_RECOVERY_ON_GET_PATH_EXCEPTION,
          null);
    }
  }

  /**
   * Redo the rename operation when path is present in atomic directory list
   * or when path has {@link RenameAtomicity#SUFFIX} suffix.
   *
   * @param path path of the pendingJson for the atomic path.
   * @param renamePendingJsonLen length of the pendingJson file.
   * @param tracingContext tracing context.
   *
   * @throws AzureBlobFileSystemException server error
   */

View on GitHub (pinned to 2add963021)

Solutions

Retry the file-system operation (rename/open) with backoff — once the transient GetPathStatus error clears, the recovery usually completes.
Check for concurrent writers finishing the same rename; verify the destination now exists before retrying.
If a job crashed mid-rename, inspect the container for leftover rename-pending JSON / atomic-rename suffix files and clean them once no writer is active.
If persistent, capture the underlying status code and request-id from logs and investigate account-side issues (throttling, firewall, soft-delete).

Example fix

// before
fs.rename(src, dst); // fails once during transient outage on non-HNS recovery
// after
boolean ok = false;
for (int i = 0; i < 3 && !ok; i++) {
  try { ok = fs.rename(src, dst); }
  catch (AbfsRestOperationException ex) {
    if (ex.getStatusCode() != 404) { throw ex; }
    sleepBackoff(i);
  }
}

Defensive patterns

Strategy: retry

Try / catch

catch (AbfsRestOperationException ex) {
  if (ex.getStatusCode() == 404
      && ex.getMessage().contains("atomic rename")) {
    // verify destination exists, then retry the rename with backoff
  } else {
    throw ex;
  }
}

Prevention

Prefer HNS-enabled accounts for rename-heavy pipelines; the non-HNS atomic-rename recovery path is what throws this.
Never run two clients renaming the same source path concurrently.
Make retry logic re-check the destination before re-issuing a failed rename.

When it happens

Trigger: A rename on a non-HNS account goes through the atomic-rename recovery flow (rename-pending JSON / RenameAtomicity SUFFIX handling), and the follow-up GetPathStatus on the source throws an AbfsRestOperationException whose status is neither 404 nor 409 (e.g. 500, 403, timeout), so renameSrcHasChanged stays false and this synthetic PATH_NOT_FOUND is thrown.

Common situations: Transient server errors or throttling during rename recovery on the blob endpoint; a proxy or middlebox returning unexpected status codes; races where another client concurrently finishes or rolls back the same rename; leftover rename-pending entries after a crashed job that is retried.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/fffacbda4c2f3b0e. Report an issue: GitHub.