MyCATApache/Mycat-Server · critical · OutOfMemoryError

error while calling spill() on

Error message

error while calling spill() on ${spiller} : ${reason}

What it means

During acquireExecutionMemory, DataNodeMemoryManager asks Spillable consumers to spill() to disk to free execution memory. If a consumer's spill() throws IOException, the manager logs the failure and throws OutOfMemoryError, because without spilling it cannot satisfy the memory request.

Solutions

1. Check the wrapped IOException in the log ('error while calling spill() on <consumer>') to find the root cause (disk full, permission, path).
2. Free disk space on the spill directories and verify they are writable by the process.
3. Point the spill/local directories at a healthy, larger disk via configuration.
4. Reduce per-task memory pressure (smaller batches, fewer concurrent consumers) or increase execution memory so spilling is not required.

Example fix

// before (spill target on a full/unwritable disk)
conf.set("mycat.local.dir", "/var/tmp");
// after
// point spill dirs at a writable disk with free space
conf.set("mycat.local.dir", "/mnt/bigdisk/spill");

// before requesting memory, verify spill dirs are usable
for (String dir : spillDirs) {
  File d = new File(dir);
  if (!d.canWrite() || d.getUsableSpace() < minSpillBytes)
    throw new IllegalStateException("spill dir unusable: " + dir);
}

try {
  long got = memoryManager.acquireExecutionMemory(required, ctx, mode);
} catch (OutOfMemoryError e) {
  // inspect cause/logging for the spill IOException; shrink workload or fail task cleanly
  logger.error("execution memory unavailable after spill failure", e);
  throw new TaskKilledException(e);
}

When it happens

Trigger: Calling acquireExecutionMemory (directly or via allocatePage/allocateLongArray on a MemoryConsumer) when execution memory is exhausted and one of the registered spillable consumers throws IOException from its spill() method (e.g. disk full, I/O error, bad temp directory).

Common situations: Disk full or disk failure on the spill directory; temp directory not writable or removed at runtime; a misconfigured spark-style local dir; large sorts/aggregations/joins exceeding available execution memory so spilling is required.

Related errors

• error while calling spill() on
• Unable to acquire bytes of memory, got
• Initial capacity exceeds maximum capacity of
• Page size cannot exceed
• Cannot allocate a page with more than

AI-assisted analysis of MyCATApache/Mycat-Server@65f8d8beb7 (2026-09-11). Data as JSON: /api/errors/9bdc58627bf6f699. Report an issue: GitHub.

Appendix: source

Thrown at src/main/java/io/mycat/memory/unsafe/memory/mm/DataNodeMemoryManager.java:148

        // Call spill() on other consumers to release memory
        for (MemoryConsumer c: consumers) {
          if (c != consumer && c.getUsed() > 0) {
            try {
               /**
               * 调用spill函数，写数据到磁盘中
               */
              long released = c.spill(required - got, consumer);
                if (released > 0 && mode == tungstenMemoryMode) {
                logger.info("Thread "+connectionAttemptId+" released "+ JavaUtils.bytesToString(released) +
                        " from "+ c +" for" + consumer);
                got += memoryManager.acquireExecutionMemory(required - got, connectionAttemptId, mode);
                if (got >= required) {
                  break;
                }
              }
            } catch (IOException e) {
              logger.error("error while calling spill() on " + c, e);
              throw new OutOfMemoryError("error while calling spill() on " + c + " : "
                + e.getMessage());
            }
          }
        }
      }

      // call spill() on itself
      if (got < required && consumer != null) {
        try {
          long released = consumer.spill(required - got, consumer);
          if (released > 0 && mode == tungstenMemoryMode) {
            logger.info("Thread " + connectionAttemptId +
                    " released "+   JavaUtils.bytesToString(released) +"from itself ("+consumer+ ")");
            got += memoryManager.acquireExecutionMemory(required - got, connectionAttemptId, mode);
          }
        } catch (IOException e) {
          logger.error("error while calling spill() on " + consumer, e);
          throw new OutOfMemoryError("error while calling spill() on " + consumer + " : "

View on GitHub (pinned to 65f8d8beb7)