apache/hadoop · error · QuotaByStorageTypeExceededException

Quota by storage type : type on path : pathName is exceeded.

Error message

Quota by storage type : type on path : pathName is exceeded. quota = long2String(quota, "B", 2) but space consumed = long2String(count, "B", 2)

What it means

QuotaByStorageTypeExceededException thrown from verifyQuotaByStorageType(), which iterates StorageType.getTypesSupportingQuota() and rejects a write when quota.getTypeSpace(t) is set and usage + typeDelta exceeds it for any type. Per-type quotas are set with 'hdfs dfsadmin -setSpaceQuota -t <TYPE> <N> <path>' and gate how many bytes of a file's blocks may live on a specific tier (DISK/SSD/ARCHIVE), as steered by the path's storage policy.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java:200

  private void verifyStoragespaceQuota(long delta) throws DSQuotaExceededException {
    if (Quota.isViolated(quota.getStorageSpace(), usage.getStorageSpace(), delta)) {
      throw new DSQuotaExceededException(quota.getStorageSpace(),
          usage.getStorageSpace() + delta);
    }
  }

  private void verifyQuotaByStorageType(EnumCounters<StorageType> typeDelta)
      throws QuotaByStorageTypeExceededException {
    if (!isQuotaByStorageTypeSet()) {
      return;
    }
    for (StorageType t: StorageType.getTypesSupportingQuota()) {
      if (!isQuotaByStorageTypeSet(t)) {
        continue;
      }
      if (Quota.isViolated(quota.getTypeSpace(t), usage.getTypeSpace(t),
          typeDelta.get(t))) {
        throw new QuotaByStorageTypeExceededException(
          quota.getTypeSpace(t), usage.getTypeSpace(t) + typeDelta.get(t), t);
      }
    }
  }

  /**
   * @throws QuotaExceededException if namespace, storagespace or storage type
   * space quota is violated after applying the deltas.
   */
  void verifyQuota(QuotaCounts counts) throws QuotaExceededException {
    verifyNamespaceQuota(counts.getNameSpace());
    verifyStoragespaceQuota(counts.getStorageSpace());
    verifyQuotaByStorageType(counts.getTypeSpaces());
  }

  boolean isQuotaSet() {
    return quota.anyNsSsCountGreaterOrEqual(0) ||
        quota.anyTypeSpaceCountGreaterOrEqual(0);

View on GitHub (pinned to 2add963021)

Solutions

1. Raise the type quota: hdfs dfsadmin -setSpaceQuota -t ARCHIVE 5T <path> (repeat per type as needed).
2. Change the storage policy so new writes target a tier with headroom: hdfs dfsadmin -setStoragePolicy -path <path> -policy HOT, then verify with -getStoragePolicy.
3. Delete or compact data on the saturated tier under the path.
4. Check current per-type usage first: hdfs dfs -count -t SSD,DISK,ARCHIVE <path>.

Example fix

# before
hdfs dfsadmin -setStoragePolicy -path /cold -policy COLD
hdfs dfsadmin -setSpaceQuota -t ARCHIVE 500G /cold   # distcp fills it
# after
hdfs dfsadmin -setSpaceQuota -t ARCHIVE 2T /cold
hdfs dfs -count -t ARCHIVE /cold

QuotaUsage q = dfs.getQuotaUsage(dir);
String policy = dfs.getStoragePolicy(dir).getName();   // e.g. COLD -> ARCHIVE
StorageType tier = policy.equals("COLD") ? StorageType.ARCHIVE : StorageType.DISK;
if (q.getTypeQuota(tier) >= 0
    && q.getTypeConsumed(tier) + bytes * replication > q.getTypeQuota(tier)) {
  // raise -setSpaceQuota -t <tier>, change policy, or shed data first
}

static boolean isTypeQuotaExceeded(IOException e) {
  return e instanceof QuotaByStorageTypeExceededException
      || (e instanceof RemoteException re
          && re.getClassName().endsWith("QuotaByStorageTypeExceededException"));
}

try {
  fs.create(dst);
} catch (RemoteException re) {
  if (re.getClassName().endsWith("QuotaByStorageTypeExceededException")) {
    // tier is full: raise the type quota or setStoragePolicy to a tier with headroom
  } else { throw re; }
}

When it happens

Trigger: Writing files under a directory whose storage policy (e.g., COLD -> ARCHIVE, ALL_SSD -> SSD) routes new blocks to a tier where the type quota is set and consumed + delta > quota; calling setStoragePolicy on a path then appending so new blocks land on the capped tier; distcp copying hot data into a COLD directory with an ARCHIVE quota.

Common situations: Tiered clusters (SSD hot tier / ARCHIVE cold tier) with per-type budgets; storage policy changed by an admin after quotas were sized; replication also multiplies per-type usage; verify current numbers with 'hdfs dfs -count -t <types> <path>' or getQuotaUsage().

Related errors

• "Failed to set quota by storage type because either" + DFS_Q
• Storage type {} is not available. Available storage types ar
• Invalid values for quota :{}
• Invalid storage type(null)
• Don't support Quota for storage type : {}

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/658a4858b3cab9e4. Report an issue: GitHub.