apache/hadoop · error · ObsException

System failure

Error message

System failure

What it means

fsRemoveKeysByDepth deletes a directory tree in batches ordered by depth (children before parents) to keep parents and children out of the same batch. It walks the status array from the end and expects directory depth to only decrease; a key that is deeper than the current depth breaks the invariant, so it warns 'The objects list is invalid because it isn't sorted by path depth.' and throws a bare ObsException('System failure'). This is an internal sanity failure, not a user-input error.

Source

Thrown at hadoop-cloud-storage-project/hadoop-huaweicloud/src/main/java/org/apache/hadoop/fs/obs/OBSPosixBucketUtils.java:607

      // Check leaf folder at current depth.
      int keyDepth = fsGetObjectKeyDepth(key);
      if (keyDepth == depth) {
        // Any key at current depth must be a leaf.
        leafKeys.add(new KeyAndVersion(key, null));
        continue;
      }
      if (keyDepth < depth) {
        // The last batch delete at current depth.
        OBSCommonUtils.removeKeys(owner, leafKeys, true, false);
        // Go on at the upper depth.
        depth = keyDepth;
        leafKeys.add(new KeyAndVersion(key, null));
        continue;
      }
      LOG.warn(
          "The objects list is invalid because it isn't sorted by"
              + " path depth.");
      throw new ObsException("System failure");
    }

    // The last batch delete at the minimum depth of all keys.
    OBSCommonUtils.removeKeys(owner, leafKeys, true, false);
  }

  // Used to create a folder
  static void fsCreateFolder(final OBSFileSystem owner,
      final String objectName)
      throws ObsException {
    for (int retryTime = 1;
        retryTime < OBSCommonUtils.MAX_RETRY_TIME; retryTime++) {
      try {
        innerFsCreateFolder(owner, objectName);
        return;
      } catch (ObsException e) {
        LOG.warn("Failed to create folder [{}], retry time [{}], "
            + "exception [{}]", objectName, retryTime, e);

View on GitHub (pinned to 2add963021)

Solutions

1. Retry the delete: transient listing-order issues usually clear on the next attempt
2. Upgrade hadoop-huaweicloud and the OBS SDK together to matched versions where ordering handling is fixed
3. Work around by deleting in smaller subtrees, or fall back to a prefix-based object delete
4. If reproducible, enable debug logging to capture the listing order and report it to the connector maintainers

Example fix

// before
fs.delete(hugeDir, true); // aborts: ObsException("System failure")

// after
// split the delete into smaller subtrees
for (FileStatus child : fs.listStatus(hugeDir)) {
  fs.delete(child.getPath(), true);
}
fs.delete(hugeDir, true);

try {
  fs.delete(dir, true);
} catch (ObsException e) {
  if ("System failure".equals(e.getStatus())) { // depth-order invariant broke
    for (FileStatus child : fs.listStatus(dir)) {
      fs.delete(child.getPath(), true); // delete per subtree
    }
    fs.delete(dir, true);
  } else {
    throw e;
  }
}

When it happens

Trigger: Deleting a very large directory tree whose listing spans many pages; bucket versioning or SDK listing behavior that returns objects in an order violating the lexicographic assumption the code relies on; mismatched OBS SDK and hadoop-huaweicloud connector versions.

Common situations: Recursive delete of huge trash or staging directories; observed after upgrading the OBS SDK or switching bucket protocol modes; rare, but it aborts the delete with an unhelpful generic message.

Related errors

• {} is not a directory
• No such file or directory: {}
• File conflicts during rename, {}
• {path}
• file conflicts: {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/03add1a6573c44c4. Report an issue: GitHub.