{"record":{"id":"0a1ab6bf99f01e81","repo":"apache/druid","slug":"segment-s-hydrant-s-already-swapped-this-cann","errorCode":null,"errorMessage":"Segment[%s] hydrant[%s] already swapped. This cannot happen.","messagePattern":"Segment\\[(.+?)\\] hydrant\\[(.+?)\\] already swapped\\. This cannot happen\\.","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java","lineNumber":1171,"sourceCode":"    final File persistDir = computePersistDir(identifier);\n    FileUtils.mkdirp(persistDir);\n\n    objectMapper.writeValue(computeIdentifierFile(identifier), identifier);\n\n    return persistDir;\n  }\n\n  /**\n   * Persists the given hydrant and returns the number of rows persisted.\n   *\n   * @param indexToPersist hydrant to persist\n   * @param identifier     the segment this hydrant is going to be part of\n   * @return the number of rows persisted\n   */\n  private int persistHydrant(FireHydrant indexToPersist, SegmentIdWithShardSpec identifier)\n  {\n    if (indexToPersist.hasSwapped()) {\n      throw new ISE(\n          \"Segment[%s] hydrant[%s] already swapped. This cannot happen.\",\n          identifier,\n          indexToPersist\n      );\n    }\n\n    log.debug(\"Segment[%s], persisting Hydrant[%s]\", identifier, indexToPersist);\n\n    try {\n      final long startTime = System.nanoTime();\n      int numRows = indexToPersist.getIndex().numRows();\n\n      // since the sink may have been persisted before it may have lost its\n      // hydrant count, we remember that value in the sinks' metadata, so we have\n      // to pull it from there....\n      SinkMetadata sm = sinksMetadata.get(identifier);\n      if (sm == null) {\n        throw new ISE(\"Sink must not be null for identifier when persisting hydrant[%s]\", identifier);","sourceCodeStart":1153,"sourceCodeEnd":1189,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java#L1153-L1189","documentation":"persistHydrant persists an in-memory hydrant to disk; a hydrant that has already been swapped to its persisted QueryableIndex must never be persisted again. This ISE guards that invariant — persisting an already-swapped hydrant would corrupt or duplicate the on-disk segment data.","triggerScenarios":"Calling the persist path on a FireHydrant whose hasSwapped() is true — i.e. internal sequencing in which persist runs after the hydrant was already swapped to a persisted segment (e.g. persist racing with sink swap/moveToThreshold).","commonSituations":"Concurrency bugs or unusual interleavings of appenderator persist/swap during batch ingestion; reused appenderator instances across jobs with residual state; custom code driving Appenderator APIs directly and calling persist twice.","solutions":["Do not call persist on a hydrant/identifier more than once per in-memory incarnation — check hasSwapped() before persisting","Stop and restart the ingestion job with a clean appenderator to clear stale state","If reproducible, file an issue with the task logs — this indicates an engine sequencing bug, not user error"],"exampleFix":"// before\npersistHydrant(indexToPersist, identifier);\n// after\nif (!indexToPersist.hasSwapped()) {\n  persistHydrant(indexToPersist, identifier);\n}","handlingStrategy":"try-catch","validationCode":"if (indexToPersist.hasSwapped()) {\n  throw new IllegalStateException(\"Refusing to persist swapped hydrant\");\n}","typeGuard":"boolean persistable(FireHydrant h) {\n  return !h.hasSwapped();\n}","tryCatchPattern":"try {\n  persistHydrant(hydrant, identifier);\n} catch (ISE e) {\n  if (e.getMessage().contains(\"already swapped\")) {\n    // state bug: recreate appenderator\n    restartJobWithCleanAppenderator();\n  }\n  throw e;\n}","preventionTips":["Persist each hydrant at most once per in-memory incarnation","Do not drive Appenderator persist/swap concurrently from custom code","Restart with a clean appenderator if this error appears — it signals engine state corruption"],"tags":["appenderator","invariant","concurrency"],"backgroundTag":"invalid-state-transition","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-17T15:17:12.973Z"}