apache/druid · error · IllegalStateException

Missing hydrant [%,d] in identifier

Error message

Missing hydrant [%,d] in identifier [%s].

What it means

When recovering persisted sinks at startup, BatchAppenderator expects persisted hydrant directories numbered contiguously from 0. If a hydrant directory's number does not equal the running count (hydrants.size()), a hydrant in the sequence is missing and the recovered segment would be incomplete, so this ISE is thrown.

Solutions

1. Recover the missing hydrant directory from backup or deep storage, or re-run the batch job to regenerate the data
2. Delete the partially persisted sink directories for this identifier and let the batch job start fresh (data must be re-ingested)
3. Audit any cleanup tooling so it never removes numbered hydrant directories within an identifier path
4. Check filesystem for lost/renamed directories if a crash occurred during persistence

List<Integer> nums = hydrantDirs.stream()
    .map(f -> Integer.parseInt(f.getName()))
    .sorted().collect(Collectors.toList());
for (int i = 0; i < nums.size(); i++) {
  if (nums.get(i) != i) throw new IllegalStateException("Gap in hydrants: " + nums);
}

boolean hydrantsContiguous(List<File> dirs) {
  List<Integer> n = dirs.stream().map(f -> Integer.parseInt(f.getName()))
      .sorted().collect(Collectors.toList());
  return IntStream.range(0, n.size()).allMatch(i -> n.get(i) == i);
}

try {
  driver.run();
} catch (ISE e) {
  if (e.getMessage().contains("Missing hydrant")) {
    restoreFromBackupOrReingest();
  }
  throw e;
}

When it happens

Trigger: Starting a BatchAppenderator over an identifier whose persist directory contains non-contiguous hydrant numbers — e.g. hydrants 0,1,3 exist but 2 is missing — usually from partial persistence, manual deletion of directories, or interrupted writes.

Common situations: Disk cleanup scripts deleting 'old-looking' numbered directories; crash between persisting hydrants; hand-copying persist data between hosts and dropping directories.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

• Problem reading persisted sinks in path
• Cannot create directory
• Cannot create tempDir
• Cannot delete temp file
• Cannot list contents of

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/8ceee84c61d0334f. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java:1064

    final File[] sinkFiles = identifierPath.listFiles(
        (dir, fileName) -> !(Ints.tryParse(fileName) == null)
    );
    if (sinkFiles == null) {
      throw new ISE("Problem reading persisted sinks in path[%s]", identifierPath);
    }

    Arrays.sort(
        sinkFiles,
        (o1, o2) -> Ints.compare(Integer.parseInt(o1.getName()), Integer.parseInt(o2.getName()))
    );

    List<FireHydrant> hydrants = new ArrayList<>();
    for (File hydrantDir : sinkFiles) {
      final int hydrantNumber = Integer.parseInt(hydrantDir.getName());

      log.debug("Loading previously persisted partial segment at [%s]", hydrantDir);
      if (hydrantNumber != hydrants.size()) {
        throw new ISE("Missing hydrant [%,d] in identifier [%s].", hydrants.size(), identifier);
      }

      hydrants.add(
          new FireHydrant(
              new QueryableIndexSegment(indexIO.loadIndex(hydrantDir), identifier.asSegmentId()),
              hydrantNumber
          )
      );
    }

    Sink retVal = new Sink(
        identifier.getInterval(),
        schema,
        identifier.getShardSpec(),
        identifier.getVersion(),
        tuningConfig.getAppendableIndexSpec(),
        tuningConfig.getMaxRowsInMemory(),
        maxBytesTuningConfig,

View on GitHub (pinned to 9b90983fd2)