apache/druid · error · IllegalStateException

Segment[ ] hydrant[ ] already swapped. This cannot happen.

Error message

Segment[%s] hydrant[%s] already swapped. This cannot happen.

What it means

persistHydrant persists an in-memory hydrant to disk; a hydrant that has already been swapped to its persisted QueryableIndex must never be persisted again. This ISE guards that invariant — persisting an already-swapped hydrant would corrupt or duplicate the on-disk segment data.

Solutions

Do not call persist on a hydrant/identifier more than once per in-memory incarnation — check hasSwapped() before persisting
Stop and restart the ingestion job with a clean appenderator to clear stale state
If reproducible, file an issue with the task logs — this indicates an engine sequencing bug, not user error

Example fix

// before
persistHydrant(indexToPersist, identifier);
// after
if (!indexToPersist.hasSwapped()) {
  persistHydrant(indexToPersist, identifier);
}

Defensive patterns

Strategy: try-catch

Validate before calling

if (indexToPersist.hasSwapped()) {
  throw new IllegalStateException("Refusing to persist swapped hydrant");
}

Type guard

boolean persistable(FireHydrant h) {
  return !h.hasSwapped();
}

Try / catch

try {
  persistHydrant(hydrant, identifier);
} catch (ISE e) {
  if (e.getMessage().contains("already swapped")) {
    // state bug: recreate appenderator
    restartJobWithCleanAppenderator();
  }
  throw e;
}

Prevention

Persist each hydrant at most once per in-memory incarnation
Do not drive Appenderator persist/swap concurrently from custom code
Restart with a clean appenderator if this error appears — it signals engine state corruption

When it happens

Trigger: Calling the persist path on a FireHydrant whose hasSwapped() is true — i.e. internal sequencing in which persist runs after the hydrant was already swapped to a persisted segment (e.g. persist racing with sink swap/moveToThreshold).

Common situations: Concurrency bugs or unusual interleavings of appenderator persist/swap during batch ingestion; reused appenderator instances across jobs with residual state; custom code driving Appenderator APIs directly and calling persist twice.

Understand the failure class

Background: "Invalid state transition" errors: "status must be X, actually Y", "already rejected/charging/uninstalled", "cannot ... while running" — what they mean when a library rejects your call — this error's family across 31 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/0a1ab6bf99f01e81. Report an issue: GitHub.

Appendix: source

Thrown at server/src/main/java/org/apache/druid/segment/realtime/appenderator/BatchAppenderator.java:1171

    final File persistDir = computePersistDir(identifier);
    FileUtils.mkdirp(persistDir);

    objectMapper.writeValue(computeIdentifierFile(identifier), identifier);

    return persistDir;
  }

  /**
   * Persists the given hydrant and returns the number of rows persisted.
   *
   * @param indexToPersist hydrant to persist
   * @param identifier     the segment this hydrant is going to be part of
   * @return the number of rows persisted
   */
  private int persistHydrant(FireHydrant indexToPersist, SegmentIdWithShardSpec identifier)
  {
    if (indexToPersist.hasSwapped()) {
      throw new ISE(
          "Segment[%s] hydrant[%s] already swapped. This cannot happen.",
          identifier,
          indexToPersist
      );
    }

    log.debug("Segment[%s], persisting Hydrant[%s]", identifier, indexToPersist);

    try {
      final long startTime = System.nanoTime();
      int numRows = indexToPersist.getIndex().numRows();

      // since the sink may have been persisted before it may have lost its
      // hydrant count, we remember that value in the sinks' metadata, so we have
      // to pull it from there....
      SinkMetadata sm = sinksMetadata.get(identifier);
      if (sm == null) {
        throw new ISE("Sink must not be null for identifier when persisting hydrant[%s]", identifier);

View on GitHub (pinned to 9b90983fd2)