{"record":{"id":"8ceee84c61d0334f","repo":"apache/druid","slug":"missing-hydrant-d-in-identifier-s","errorCode":null,"errorMessage":"Missing hydrant [%,d] in identifier [%s].","messagePattern":"Missing hydrant \\[%,d\\] in identifier \\[(.+?)\\]\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java","lineNumber":1064,"sourceCode":"    final File[] sinkFiles = identifierPath.listFiles(\n        (dir, fileName) -> !(Ints.tryParse(fileName) == null)\n    );\n    if (sinkFiles == null) {\n      throw new ISE(\"Problem reading persisted sinks in path[%s]\", identifierPath);\n    }\n\n    Arrays.sort(\n        sinkFiles,\n        (o1, o2) -> Ints.compare(Integer.parseInt(o1.getName()), Integer.parseInt(o2.getName()))\n    );\n\n    List<FireHydrant> hydrants = new ArrayList<>();\n    for (File hydrantDir : sinkFiles) {\n      final int hydrantNumber = Integer.parseInt(hydrantDir.getName());\n\n      log.debug(\"Loading previously persisted partial segment at [%s]\", hydrantDir);\n      if (hydrantNumber != hydrants.size()) {\n        throw new ISE(\"Missing hydrant [%,d] in identifier [%s].\", hydrants.size(), identifier);\n      }\n\n      hydrants.add(\n          new FireHydrant(\n              new QueryableIndexSegment(indexIO.loadIndex(hydrantDir), identifier.asSegmentId()),\n              hydrantNumber\n          )\n      );\n    }\n\n    Sink retVal = new Sink(\n        identifier.getInterval(),\n        schema,\n        identifier.getShardSpec(),\n        identifier.getVersion(),\n        tuningConfig.getAppendableIndexSpec(),\n        tuningConfig.getMaxRowsInMemory(),\n        maxBytesTuningConfig,","sourceCodeStart":1046,"sourceCodeEnd":1082,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java#L1046-L1082","documentation":"When recovering persisted sinks at startup, BatchAppenderator expects persisted hydrant directories numbered contiguously from 0. If a hydrant directory's number does not equal the running count (hydrants.size()), a hydrant in the sequence is missing and the recovered segment would be incomplete, so this ISE is thrown.","triggerScenarios":"Starting a BatchAppenderator over an identifier whose persist directory contains non-contiguous hydrant numbers — e.g. hydrants 0,1,3 exist but 2 is missing — usually from partial persistence, manual deletion of directories, or interrupted writes.","commonSituations":"Disk cleanup scripts deleting 'old-looking' numbered directories; crash between persisting hydrants; hand-copying persist data between hosts and dropping directories.","solutions":["Recover the missing hydrant directory from backup or deep storage, or re-run the batch job to regenerate the data","Delete the partially persisted sink directories for this identifier and let the batch job start fresh (data must be re-ingested)","Audit any cleanup tooling so it never removes numbered hydrant directories within an identifier path","Check filesystem for lost/renamed directories if a crash occurred during persistence"],"exampleFix":null,"handlingStrategy":"validation","validationCode":"List<Integer> nums = hydrantDirs.stream()\n    .map(f -> Integer.parseInt(f.getName()))\n    .sorted().collect(Collectors.toList());\nfor (int i = 0; i < nums.size(); i++) {\n  if (nums.get(i) != i) throw new IllegalStateException(\"Gap in hydrants: \" + nums);\n}","typeGuard":"boolean hydrantsContiguous(List<File> dirs) {\n  List<Integer> n = dirs.stream().map(f -> Integer.parseInt(f.getName()))\n      .sorted().collect(Collectors.toList());\n  return IntStream.range(0, n.size()).allMatch(i -> n.get(i) == i);\n}","tryCatchPattern":"try {\n  driver.run();\n} catch (ISE e) {\n  if (e.getMessage().contains(\"Missing hydrant\")) {\n    restoreFromBackupOrReingest();\n  }\n  throw e;\n}","preventionTips":["Never manually delete numbered hydrant directories inside an identifier path","Persist in a crash-safe way so partial writes do not lose hydrants","Back up persist directories before maintenance/cleanup"],"tags":["filesystem","persist","data-loss"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}