apache/druid · error · SegmentLoadingException

Failed to load segment[%s] in all locations.

Error message

Failed to load segment[%s] in all locations.

What it means

Thrown after the manager has attempted to load a segment in every configured cache location and each attempt failed with a SegmentLoadingException. It is the terminal error aggregating per-location failures (each logged as 'Failed to load segment[%s] in location[%s], trying next location').

Source

Thrown at server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:2020

    final Iterator<StorageLocation> locationsIterator = strategy.getLocations();
    while (locationsIterator.hasNext()) {
      final StorageLocation location = locationsIterator.next();
      if (location.reserve(cacheEntry)) {
        try {
          final CompleteSegmentCacheEntry entry = checkComplete(location.getCacheEntry(cacheEntry.id), cacheEntry.id);
          if (entry != null) {
            entry.lazyLoadCallback = segmentLoadFailCallback;
            entry.setOnUnmount(null);
            entry.mount(location);
            return entry;
          }
        }
        catch (SegmentLoadingException e) {
          log.warn(e, "Failed to load segment[%s] in location[%s], trying next location", cacheEntry.id, location.getPath());
        }
      }
    }
    throw new SegmentLoadingException("Failed to load segment[%s] in all locations.", cacheEntry.id);
  }

  /**
   * Narrow a {@link CacheEntry} to {@link CompleteSegmentCacheEntry} with a defensive check. Returns null when the
   * entry is missing. Throws when a non-complete entry (e.g. {@link PartialSegmentMetadataCacheEntry}) is registered
   * under the same identifier. Callers of {@link #assignLocationAndMount} only operate on the non-virtual-storage
   * eager path, so encountering a partial entry here is a programming error.
   */
  @Nullable
  private static CompleteSegmentCacheEntry checkComplete(@Nullable CacheEntry entry, SegmentCacheEntryIdentifier id)
  {
    if (entry == null) {
      return null;
    }
    if (entry instanceof CompleteSegmentCacheEntry complete) {
      return complete;
    }
    throw DruidException.defensive(

View on GitHub (pinned to 9b90983fd2)

Solutions

1. Read the per-location warnings logged immediately before this error to find the root cause (they carry the real exception).
2. Check deep storage connectivity and credentials from the historical node.
3. Verify cache locations have free space and write permissions; run df and touch a test file in each location path.
4. Retry the load via coordinator; if the segment is corrupt in deep storage, re-ingest or restore from backup.

try { loadSegment(segmentId); } catch (SegmentLoadingException e) { // per-location warnings logged above carry root cause; trigger coordinator re-load or mark segment failed }

When it happens

Trigger: Any path calling assignLocationAndMount/multi-location load where every location's load attempt throws SegmentLoadingException (download failure, unparseable segment, no space, all locations unhealthy).

Common situations: All cache volumes full or unwritable; deep storage unreachable so downloads fail at each location; corrupt segment files in deep storage; all locations' disks failing.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

• Failed to load segment[%s] in reserved location[%s]
• e.getMessage()
• Do not know how to handle file type at [%s]
• Error loading [%s]
• Unknown index version[%s]

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/1aa91bf2c39e9065. Report an issue: GitHub.