MyCATApache/Mycat-Server · critical · OutOfMemoryError

error while calling spill() on

Error message

error while calling spill() on ${consumer} : ${reason}

What it means

In acquireExecutionMemory, when the requested memory cannot be obtained directly, DataNodeMemoryManager calls the consumer's own spill() to free space. If that spill() call throws IOException, the manager wraps it into an OutOfMemoryError stating that spill failed on that consumer, since memory cannot be freed.

Solutions

1. Inspect the chained IOException cause in logs to identify the spill I/O failure (ENOSPC, EACCES, missing dir).
2. Free space and ensure write permissions on the spill directory.
3. Reconfigure spill directories to reliable storage with sufficient free space.
4. Reduce memory demand (lower concurrency, smaller page sizes/batches) or raise the execution memory limit.

Example fix

// before
long got = memoryManager.acquireExecutionMemory(required, taskContext, mode);
// after
// guard memory pressure and spill health beforehand
if (diskUsableBytes(spillDir) < minSpillSpace) {
  throw new IllegalStateException("spill dir too small: " + spillDir);
}
long got = memoryManager.acquireExecutionMemory(required, taskContext, mode);

// ensure the consumer's spill target is healthy before acquiring
if (!spillDir.canWrite() || spillDir.getUsableSpace() < minSpillBytes) {
  throw new IllegalStateException("cannot spill to " + spillDir);
}

try {
  long got = memoryManager.acquireExecutionMemory(required, attemptId, mode);
} catch (OutOfMemoryError e) {
  // consumer spill() threw IOException; log and retry with less demand or kill task
  logger.error("spill failed while acquiring execution memory", e);
  throw new TaskKilledException(e);
}

When it happens

Trigger: A MemoryConsumer requests execution memory (acquireExecutionMemory or allocatePage/allocateArray on it) while under memory pressure, and the consumer's own spill() implementation throws IOException.

Common situations: Sorters/aggregators spilling to a full, failed, or unwritable disk; temp files deleted by a cleaner job mid-run; heavy concurrent workloads forcing frequent spills on marginal storage.

Related errors

• error while calling spill() on
• Unable to acquire bytes of memory, got
• Initial capacity exceeds maximum capacity of
• Page size cannot exceed
• Cannot allocate a page with more than

AI-assisted analysis of MyCATApache/Mycat-Server@65f8d8beb7 (2026-09-11). Data as JSON: /api/errors/8194851ed50edc8d. Report an issue: GitHub.

Appendix: source

Thrown at src/main/java/io/mycat/memory/unsafe/memory/mm/DataNodeMemoryManager.java:166

              throw new OutOfMemoryError("error while calling spill() on " + c + " : "
                + e.getMessage());
            }
          }
        }
      }

      // call spill() on itself
      if (got < required && consumer != null) {
        try {
          long released = consumer.spill(required - got, consumer);
          if (released > 0 && mode == tungstenMemoryMode) {
            logger.info("Thread " + connectionAttemptId +
                    " released "+   JavaUtils.bytesToString(released) +"from itself ("+consumer+ ")");
            got += memoryManager.acquireExecutionMemory(required - got, connectionAttemptId, mode);
          }
        } catch (IOException e) {
          logger.error("error while calling spill() on " + consumer, e);
          throw new OutOfMemoryError("error while calling spill() on " + consumer + " : "
            + e.getMessage());

        }
      }

      if (consumer != null) {
        consumers.add(consumer);
      }
     // logger.info("Thread" + connectionAttemptId + " acquire "+  JavaUtils.bytesToString(got) +" for "+ consumer+"");
      return got;
    }
  }

  /**
   * Release N bytes of execution memory for a MemoryConsumer.
   */
  public void releaseExecutionMemory(long size, MemoryMode mode, MemoryConsumer consumer) {
    logger.debug   ("Thread" + connectionAttemptId + " release "+  JavaUtils.bytesToString(size) +" from "+ consumer+"");

View on GitHub (pinned to 65f8d8beb7)