apache/druid · error · IllegalStateException

Problem reading persisted sinks in path

Error message

Problem reading persisted sinks in path[%s]

What it means

When a BatchAppenderator is created, it scans the identifier's persist directory for previously persisted sinks, filtering to numeric-named subdirectories (hydrant numbers). File.listFiles() returning null means the path is not a readable directory (or an I/O error occurred), so Druid throws this ISE instead of silently starting with no data.

Solutions

1. Check that the path in the message exists, is a directory, and is readable/writable by the Druid process user
2. Fix file ownership/permissions on the persist directory (druid.segment.cacheInfoDir / intermediate persist dirs)
3. Remove or repair the corrupted identifier directory if the data can be re-ingested
4. Check for storage-layer (NFS/HDFS/mount) errors in preceding logs

File dir = new File(identifierPath);
if (!dir.isDirectory() || !dir.canRead()) {
  throw new IllegalStateException("Persist dir not readable: " + dir);
}

boolean isReadableDirectory(File f) {
  return f != null && f.isDirectory() && f.canRead();
}

try {
  appenderator.startJob();
} catch (ISE e) {
  if (e.getMessage().contains("Problem reading persisted sinks")) {
    repairOrCleanPersistDir(path);
  }
  throw e;
}

When it happens

Trigger: Constructing a BatchAppenderator whose segment identifier directory exists but cannot be listed — e.g. permission problems on the base persist dir, the path being a regular file instead of a directory, or underlying storage errors (NFS/object mount) during listFiles.

Common situations: Corrupted or hand-edited deep-storage/persist layout; restored data with wrong ownership/permissions; base-suffix persist path conflicts between tasks; mounting changes after restart.

Related errors

• Missing hydrant [%,d] in identifier
• Can't find previous segmentIds for sequence
• Can't find previously allocated segmentId
• Cannot create directory
• Cannot create tempDir

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8962631473b2bb21. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java:1050

      if (!identifierFile.isFile()) {
        // No identifier in this sinkDir; it must not actually be a sink directory. Skip it.
        continue;
      }
      retVal.add(sinkDir);
    }

    return retVal;
  }

  private Sink getSinkForIdentifierPath(SegmentIdWithShardSpec identifier, File identifierPath)
      throws IOException
  {
    // To avoid reading and listing of "merged" dir and other special files
    final File[] sinkFiles = identifierPath.listFiles(
        (dir, fileName) -> !(Ints.tryParse(fileName) == null)
    );
    if (sinkFiles == null) {
      throw new ISE("Problem reading persisted sinks in path[%s]", identifierPath);
    }

    Arrays.sort(
        sinkFiles,
        (o1, o2) -> Ints.compare(Integer.parseInt(o1.getName()), Integer.parseInt(o2.getName()))
    );

    List<FireHydrant> hydrants = new ArrayList<>();
    for (File hydrantDir : sinkFiles) {
      final int hydrantNumber = Integer.parseInt(hydrantDir.getName());

      log.debug("Loading previously persisted partial segment at [%s]", hydrantDir);
      if (hydrantNumber != hydrants.size()) {
        throw new ISE("Missing hydrant [%,d] in identifier [%s].", hydrants.size(), identifier);
      }

      hydrants.add(
          new FireHydrant(

View on GitHub (pinned to 9b90983fd2)