apache/druid · warning

Failed to run deferred eviction of container

Error message

Failed to run deferred eviction of container[%d] for [%s]

What it means

During deferred eviction (unmapping and deleting the local container file) run in fetchRun's finally block, evictContainer threw. Because this runs during cleanup, the exception is only logged as a warning and swallowed so it does not mask the original fetch outcome. Failure here can leave a container view mapped or its local file present.

Solutions

1. Check the logged Throwable for the eviction root cause (I/O vs mapping failure)
2. Verify disk and permissions on the local segment cache directory
3. Restart the node if container mappings leak (check memory-mapped regions)
4. Retry the fetch; eviction is retried on subsequent runs

Example fix

null

// Ensure cache directory exists and is writable before enabling containerized fetch
Files.isDirectory(cacheDir) && Files.isWritable(cacheDir);

null

// Deferred eviction is non-throwing by design; monitor WARN logs and rely on retry fetch
// Treat as best-effort cleanup — no caller action required

When it happens

Trigger: evictContainer (which unmaps the in-memory view and deletes local files) throws during deferred eviction after a fetch, e.g. munmap failure, file lock, or I/O error deleting the local file.

Common situations: Disk issues on the segment cache directory, concurrent access to the container file, resource exhaustion preventing unmapping on the host.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Could not close log file for
• druid.segmentCache.locations must be set on historicals.
• Failed to get segment factory for
• Failed to load segment
• Failed to load segment

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/b86d0aedda594cff. Report an issue: GitHub.

Appendix: source

Thrown at processing/src/main/java/org/apache/druid/segment/file/PartialSegmentFileMapperV10.java:1031

    boolean evictNow = false;
    containerLocks[containerIndex].lock();
    try {
      containerFetchesInFlight[containerIndex]--;
      if (containerFetchesInFlight[containerIndex] == 0 && containerEvictionPending[containerIndex]) {
        containerEvictionPending[containerIndex] = false;
        evictNow = true;
      }
    }
    finally {
      containerLocks[containerIndex].unlock();
    }
    if (evictNow) {
      // Runs from fetchRun's finally, so it must not throw over whatever brought us here.
      try {
        evictContainer(containerIndex);
      }
      catch (Throwable t) {
        LOG.warn(t, "Failed to run deferred eviction of container[%d] for [%s]", containerIndex, targetFilename);
      }
    }
  }

  /**
   * Reverse of {@link #initializeContainer(int)}: unmap the in-memory view of the container, delete the local
   * container file, and clear the bitmap bits + {@link #downloadedFiles} entries for every internal file that lived
   * in this container.
   * <p>
   * <b>Concurrency contract.</b> No concurrent {@link #mapFile} may be in flight for any file in this container;
   * that is enforced one layer up by the cache-entry refcount, since {@code PartialSegmentBundleCacheEntry} evicts
   * from its {@code doActualUnmount} callback, which fires only after every reference acquired via
   * {@code acquireReference()} has been closed. Bypassing that gate is dangerous: {@link ByteBufferUtils#unmap} frees
   * the off-heap mapping, so a {@link ByteBuffer#slice} from a concurrent reader is a JVM SIGSEGV, not a recoverable
   * error. An in-flight {@link #fetchFiles}/{@link #fetchRun} is handled here instead of by the caller: the eviction
   * is deferred to whichever fetch finishes last, because callers can hold the storage location's write lock and must
   * not block on a deep-storage read.
   * <p>

View on GitHub (pinned to 9b90983fd2)