apache/druid · error · SegmentLoadingException

Was not able to create new download marker for

Error message

Was not able to create new download marker for [%s]

What it means

Thrown when creating the DOWNLOAD_START_MARKER file inside the segment's storage directory fails because File.createNewFile() returned false, meaning a file already exists at that path (typically a stale marker left by an earlier interrupted download). The download cannot safely proceed without a fresh marker.

Solutions

1. Delete the stale downloadStartMarker (and partial directory) for the segment under the cache location, then retry the load.
2. Check for concurrent loads of the same segment on this host; ensure only one historical/worker owns the segment.
3. Restart the historical so the startup cleanup removes orphaned markers and partial dirs.
4. Verify cache directory permissions allow create/delete operations.

try { loadSegment(segment); } catch (SegmentLoadingException e) { if (e.getMessage().contains("create new download marker")) { deleteStaleMarkerAndRetry(segment); } }

When it happens

Trigger: loadInLocation path where new File(storageDir, DOWNLOAD_START_MARKER_FILE_NAME).createNewFile() returns false — the marker already exists in storageDir, e.g. leftover from a crashed previous download of the same segment.

Common situations: Historical restart after unclean shutdown with orphaned download markers; concurrent loads of the same segment racing into the same directory; leftover markers not cleaned by startup sweep.

Understand the failure class

Background: "already exists" / EEXIST / FileAlreadyExistsException: what the 'file already exists' error means and how to fix it — this error's family across 37 libraries.

Related errors

• Unable to remove marker file for
• druid.segmentCache.locations must be set on historicals.
• Failed to get segment factory for
• Failed to load segment
• Failed to load segment

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/26f3d47aad6340c3. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java:2418

    }

    public File toPotentialLocation(final File location)
    {
      return new File(location, relativePathString);
    }

    @GuardedBy("entryLock")
    private void loadInLocationWithStartMarker(final DataSegment segment, final File storageDir)
        throws SegmentLoadingException
    {
      // We use a marker to prevent the case where a segment is downloaded, but before the download completes,
      // the parent directories of the segment are removed
      final File downloadStartMarker = new File(storageDir, DOWNLOAD_START_MARKER_FILE_NAME);
      try {
        FileUtils.mkdirp(storageDir);

        if (!downloadStartMarker.createNewFile()) {
          throw new SegmentLoadingException("Was not able to create new download marker for [%s]", storageDir);
        }
        loadInLocation(segment, storageDir);

        if (!downloadStartMarker.delete()) {
          throw new SegmentLoadingException("Unable to remove marker file for [%s]", storageDir);
        }
      }
      catch (IOException e) {
        throw new SegmentLoadingException(e, "Unable to create marker file for [%s]", storageDir);
      }
    }

    @GuardedBy("entryLock")
    private void loadInLocation(final DataSegment segment, final File storageDir)
        throws SegmentLoadingException
    {
      // LoadSpec isn't materialized until here so that any system can interpret Segment without having to have all the
      // LoadSpec dependencies.

View on GitHub (pinned to 9b90983fd2)