apache/druid · critical · MSQException

NotEnoughTemporaryStorageFault

Error message

NotEnoughTemporaryStorageFault

What it means

WorkerStorageParameters.createInstance checks that the usable temporary storage derived from tmpStorageBytesPerTask meets MINIMUM_SUPER_SORTER_TMP_STORAGE_BYTES; if not, it throws MSQException(NotEnoughTemporaryStorageFault) with the suggested minimum and the configured value. This guards fault-tolerant MSQ queries that spill super-sorter data to local disk but have been allocated too little temporary space.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/WorkerStorageParameters.java:108

   * @param tmpStorageBytesPerTask                  amount of disk space to be allocated per task for intermediate files.
   * @param isIntermediateSuperSorterStorageEnabled whether intermediate super sorter storage is enabled
   */
  public static WorkerStorageParameters createInstance(
      final long tmpStorageBytesPerTask,
      final boolean isIntermediateSuperSorterStorageEnabled
  )
  {
    if (!isIntermediateSuperSorterStorageEnabled || tmpStorageBytesPerTask == -1) {
      return new WorkerStorageParameters(-1);
    }

    Preconditions.checkArgument(tmpStorageBytesPerTask > 0, "Temporary storage bytes passed: [%s] should be > 0", tmpStorageBytesPerTask);
    long intermediateSuperSorterStorageMaxLocalBytes = computeUsableStorage(tmpStorageBytesPerTask);

    log.info("Intermediate super sorter local storage size: %d bytes", intermediateSuperSorterStorageMaxLocalBytes);

    if (intermediateSuperSorterStorageMaxLocalBytes < MINIMUM_SUPER_SORTER_TMP_STORAGE_BYTES) {
      throw new MSQException(
          new NotEnoughTemporaryStorageFault(
              calculateSuggestedMinTemporaryStorage(),
              tmpStorageBytesPerTask
          )
      );
    }

    return new WorkerStorageParameters(intermediateSuperSorterStorageMaxLocalBytes);
  }

  private static long computeUsableStorage(long tmpStorageBytesPerTask)
  {
    return (long) (SUPER_SORTER_TMP_STORAGE_USABLE_FRACTION * (tmpStorageBytesPerTask - MINIMUM_BASIC_OPERATIONS_BYTES));
  }

  private static long calculateSuggestedMinTemporaryStorage()
  {
    return MINIMUM_BASIC_OPERATIONS_BYTES + (long) (MINIMUM_SUPER_SORTER_TMP_STORAGE_BYTES / SUPER_SORTER_TMP_STORAGE_USABLE_FRACTION);

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Raise tmpStorageBytesPerTask to at least the value in NotEnoughTemporaryStorageFault.suggestedMinTemporaryStorage
2. Provision larger local scratch disk volumes for the peon/indexer tasks
3. Disable fault tolerance / durable shuffle if spilling is not needed for this query
4. Point the task temporary directory at a larger mount

Example fix

// before: task context
// {"druid.msq.indexer.tmp.storage.bytesPerTask": 1000000}
// after
// {"druid.msq.indexer.tmp.storage.bytesPerTask": 10000000000} // >= suggested minimum

long usable = computeUsableStorage(tmpStorageBytesPerTask);
if (usable < MINIMUM_SUPER_SORTER_TMP_STORAGE_BYTES) {
  throw new IllegalArgumentException("Raise tmpStorageBytesPerTask to >= " + calculateSuggestedMinTemporaryStorage());
}

try {
  enableFaultTolerance(query);
} catch (MSQException e) {
  if (e.getFault() instanceof NotEnoughTemporaryStorageFault f) {
    tmpStorageBytesPerTask = f.getSuggestedMinTemporaryStorage(); // retry with the suggestion
  } else { throw e; }
}

When it happens

Trigger: Fault tolerance / durable shuffle enabled with tmpStorageBytesPerTask configured below the minimum super-sorter requirement; operator sets a small per-task temp storage quota in task context or worker config; druid.indexer task tmp dir volume is tiny.

Common situations: Enabling durable shuffle storage on workers with small local disks; setting per-task temp storage limits to protect shared disks but undersizing them; running on containers with small ephemeral storage mounts.

Related errors

• Error while cleaning up temporary files at path[%s]. Skippin
• Error while cleaning up durable storage path[%s].
• Failed to get object summaries from S3 bucket[%s], prefix[%s
• Can't find location to handle segment[%s]
• Unknown

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/1d29c7f46ced37dd. Report an issue: GitHub.