{"record":{"id":"bc137068ce7cbbc6","repo":"apache/druid","slug":"unable-to-remove-marker-file-for-s","errorCode":null,"errorMessage":"Unable to remove marker file for [%s]","messagePattern":"Unable to remove marker file for \\[(.+?)\\]","errorType":"exception","errorClass":"SegmentLoadingException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java","lineNumber":2423,"sourceCode":"    }\n\n    @GuardedBy(\"entryLock\")\n    private void loadInLocationWithStartMarker(final DataSegment segment, final File storageDir)\n        throws SegmentLoadingException\n    {\n      // We use a marker to prevent the case where a segment is downloaded, but before the download completes,\n      // the parent directories of the segment are removed\n      final File downloadStartMarker = new File(storageDir, DOWNLOAD_START_MARKER_FILE_NAME);\n      try {\n        FileUtils.mkdirp(storageDir);\n\n        if (!downloadStartMarker.createNewFile()) {\n          throw new SegmentLoadingException(\"Was not able to create new download marker for [%s]\", storageDir);\n        }\n        loadInLocation(segment, storageDir);\n\n        if (!downloadStartMarker.delete()) {\n          throw new SegmentLoadingException(\"Unable to remove marker file for [%s]\", storageDir);\n        }\n      }\n      catch (IOException e) {\n        throw new SegmentLoadingException(e, \"Unable to create marker file for [%s]\", storageDir);\n      }\n    }\n\n    @GuardedBy(\"entryLock\")\n    private void loadInLocation(final DataSegment segment, final File storageDir)\n        throws SegmentLoadingException\n    {\n      // LoadSpec isn't materialized until here so that any system can interpret Segment without having to have all the\n      // LoadSpec dependencies.\n      final LoadSpec loadSpec = jsonMapper.convertValue(segment.getLoadSpec(), LoadSpec.class);\n      final LoadSpec.LoadSpecResult result = loadSpec.loadSegment(storageDir);\n      if (result.getSize() != segment.getSize()) {\n        log.warn(\n            \"Segment [%s] is different than expected size. Expected [%d] found [%d]\",","sourceCodeStart":2405,"sourceCodeEnd":2441,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/segment/loading/SegmentLocalCacheManager.java#L2405-L2441","documentation":"Thrown after a successful download when downloadStartMarker.delete() returns false, meaning the marker file could not be removed from disk. Because the marker's presence signals an incomplete download, failing to delete it leaves the segment looking partial and it may be reclaimed by later sweeps.","triggerScenarios":"loadInLocation completes and loadInLocation(segment, storageDir) succeeds, but File.delete() on the marker returns false — usually filesystem-level causes (permission change, directory removed under the manager, immutable/readonly mount).","commonSituations":"Cache volume switched to read-only or full; external cleanup job deleting files concurrently; permission mismatches after running as a different user.","solutions":["Check write/delete permissions on the cache location directory and its contents for the Druid process user.","Ensure the filesystem is not read-only or out of inodes/space (df -i, mount output).","Stop external cleanup scripts from touching druid cache directories.","Retry the load; if the marker persists, delete it manually and re-load the segment."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"// before loading, verify location is writable\ndPath.toFile().setWritable(true) && Files.isWritable(locationPath) || throw new IllegalStateException(\"cache location not writable\")","typeGuard":null,"tryCatchPattern":"try { loadSegment(segment); } catch (SegmentLoadingException e) { if (e.getMessage().contains(\"Unable to remove marker file\")) { checkMountReadOnlyAndAlert(); } }","preventionTips":["Keep cache volumes mounted read-write and monitor mount flags.","Exclude druid cache directories from external cleanup jobs.","Run all Druid processes as the same user owning the cache directories."],"tags":["segment-cache","download-marker","file-delete-failed","permissions"],"backgroundTag":"file-write-failed","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}