apache/hadoop · error · DSQuotaExceededException

The DiskSpace quota of pathName is exceeded: quota = quota B

Error message

The DiskSpace quota of pathName is exceeded: quota = quota B = long2String(quota, "B", 2) but diskspace consumed = count B = long2String(count, "B", 2)

What it means

DSQuotaExceededException thrown from verifyStoragespaceQuota(): the storage-space quota (bytes, set with 'hdfs dfsadmin -setSpaceQuota') would be exceeded after applying delta, i.e. Quota.isViolated(quota.getStorageSpace(), usage.getStorageSpace(), delta). The check runs on every write that grows disk usage; crucially HDFS counts bytes multiplied by replication factor.

Source

Thrown at hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DirectoryWithQuotaFeature.java:184

    return new QuotaCounts.Builder().quotaCount(quota).build();
  }

  /** @return the namespace and storagespace and typespace consumed. */
  public QuotaCounts getSpaceConsumed() {
    return new QuotaCounts.Builder().quotaCount(usage).build();
  }

  /** Verify if the namespace quota is violated after applying delta. */
  private void verifyNamespaceQuota(long delta) throws NSQuotaExceededException {
    if (Quota.isViolated(quota.getNameSpace(), usage.getNameSpace(), delta)) {
      throw new NSQuotaExceededException(quota.getNameSpace(),
          usage.getNameSpace() + delta);
    }
  }
  /** Verify if the storagespace quota is violated after applying delta. */
  private void verifyStoragespaceQuota(long delta) throws DSQuotaExceededException {
    if (Quota.isViolated(quota.getStorageSpace(), usage.getStorageSpace(), delta)) {
      throw new DSQuotaExceededException(quota.getStorageSpace(),
          usage.getStorageSpace() + delta);
    }
  }

  private void verifyQuotaByStorageType(EnumCounters<StorageType> typeDelta)
      throws QuotaByStorageTypeExceededException {
    if (!isQuotaByStorageTypeSet()) {
      return;
    }
    for (StorageType t: StorageType.getTypesSupportingQuota()) {
      if (!isQuotaByStorageTypeSet(t)) {
        continue;
      }
      if (Quota.isViolated(quota.getTypeSpace(t), usage.getTypeSpace(t),
          typeDelta.get(t))) {
        throw new QuotaByStorageTypeExceededException(
          quota.getTypeSpace(t), usage.getTypeSpace(t) + typeDelta.get(t), t);
      }

View on GitHub (pinned to 2add963021)

Solutions

1. Raise or clear the space quota: hdfs dfsadmin -setSpaceQuota 500G <path> or hdfs dfsadmin -clrSpaceQuota <path>.
2. Lower replication under the path: hdfs dfs -setrep -w 2 <path> (frees 1x of usage from the quota accounting).
3. Delete or archive data under the path until usage < quota.
4. Budget writes in advance: needed = bytes * replication; compare against hdfs dfs -count -q output (SPACE_QUOTA, REMAINING_SPACE_QUOTA).

Example fix

# before
hdfs dfsadmin -setSpaceQuota 1T /data/warehouse   # writes replicate x3
# after: budget 3x, or lower replication
hdfs dfsadmin -setSpaceQuota 3T /data/warehouse
hdfs dfs -setrep -w 2 /data/warehouse/cold

ContentSummary cs = dfs.getContentSummary(dir);
short rep = fs.getFileStatus(dst).getReplication();
long need = bytesToWrite * rep;                     // replication multiplies usage
if (cs.getSpaceQuota() >= 0 && cs.getSpaceConsumed() + need > cs.getSpaceQuota()) {
  // raise -setSpaceQuota, lower replication, or shed data BEFORE writing
}

static boolean isSpaceQuotaExceeded(IOException e) {
  return e instanceof DSQuotaExceededException
      || (e instanceof RemoteException re
          && re.getClassName().endsWith("DSQuotaExceededException"));
}

try {
  out = fs.create(dst);
} catch (RemoteException re) {
  if (re.getClassName().endsWith("DSQuotaExceededException")) {
    // stop producing, raise space quota or reduce replication, resume from last checkpoint
  } else { throw re; }
}

When it happens

Trigger: create()/addBlock/appending a block under a space-quota'd directory: the delta is blockReplication * blockSize (default 3x block size per block allocated); rename of a data-bearing subtree into the quota'd tree; setrep raising usage retroactively on later writes.

Common situations: Quota sized to raw bytes while default replication x3 triples consumption; raising replication on existing files inside a quota'd area; large sequential writers hitting the ceiling mid-job; distcp preserving replication into a quota'd destination.

Related errors

• The NameSpace quota (directories and files) of directory pat
• "{}" is not a valid value for a quota.
• Storage type {} is not available. Available storage types ar
• Invalid values for quota : {} and {}
• Unexpected configuration parameters: dfs.namenode.replicatio

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/462768889c39f26b. Report an issue: GitHub.