apache/druid · error · ISE

All atomicUpdateGroup must be compacted together. Expected s

Error message

All atomicUpdateGroup must be compacted together. Expected size[%s] but current size[%s]

What it means

Thrown when a completed atomicUpdateGroup (a group of consecutive segments sharing one rootPartitionRange) is closed at a range boundary but the number of segments accumulated for that group does not equal the group's declared atomicUpdateGroupSize. Every atomicUpdateGroup must be fully present in the input, because partial replacement of a group cannot produce output that safely overshadows the existing segments.

Source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/common/task/TaskLockHelper.java:296

        if (curSegment.getMinorVersion() != nextSegment.getMinorVersion()
            || curSegment.getAtomicUpdateGroupSize() != nextSegment.getAtomicUpdateGroupSize()) {
          throw new ISE(
              "segment[%s] and segment[%s] have the same rootPartitionRange, but different minorVersion or atomicUpdateGroupSize",
              curSegment,
              nextSegment
          );
        }
        atomicUpdateGroupSize++;
      } else {
        if (curSegment.getEndRootPartitionId() != nextSegment.getStartRootPartitionId()) {
          throw new ISE(
              "Can't compact segments of non-consecutive rootPartition range. Missing partitionIds between [%s] and [%s]",
              curSegment.getEndRootPartitionId(),
              nextSegment.getStartRootPartitionId()
          );
        }
        if (atomicUpdateGroupSize != curSegment.getAtomicUpdateGroupSize()) {
          throw new ISE(
              "All atomicUpdateGroup must be compacted together. Expected size[%s] but current size[%s]",
              curSegment.getAtomicUpdateGroupSize(),
              atomicUpdateGroupSize
          );
        }
        atomicUpdateGroupSize = 1;
      }
    }
    if (atomicUpdateGroupSize != sortedSegments.get(sortedSegments.size() - 1).getAtomicUpdateGroupSize()) {
      throw new ISE(
          "All atomicUpdateGroup must be compacted together. Expected size[%s] but current size[%s]",
          sortedSegments.get(sortedSegments.size() - 1).getAtomicUpdateGroupSize(),
          atomicUpdateGroupSize
      );
    }
  }
}

View on GitHub (pinned to 9b90983fd2)

Solutions

Ensure the input includes all segments of each atomicUpdateGroup (match count to the declared atomicUpdateGroupSize).
Find the missing segment(s) via the segments table (same interval and rootPartitionRange) and include or restore them.
If the group cannot be completed, re-run a full replace of the interval to regenerate a consistent set of segments.
Avoid editing/excluding individual segments of a multi-segment atomicUpdateGroup; operate at the whole-group or whole-interval level.

Example fix

// before: one shard of a size-3 group filtered out
List<DataSegment> input = all.stream().filter(s -> s.getShardSpec().getPartitionNum() != 2).collect(toList());
verifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull(input); // ISE: expected 3, got 2

// after: keep the whole group
verifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull(all);

Defensive patterns

Strategy: validation

Validate before calling

// Java: verify every atomicUpdateGroup is fully present before compaction
Map<Integer, List<DataSegment>> byStart = segments.stream()
    .collect(Collectors.groupingBy(DataSegment::getStartRootPartitionId));
byStart.forEach((start, group) -> {
  int declared = group.get(0).getAtomicUpdateGroupSize();
  if (group.size() != declared) {
    throw new IllegalStateException("Incomplete atomicUpdateGroup at rootPartition " + start
        + ": expected " + declared + ", found " + group.size());
  }
});

Try / catch

try {
  TaskLockHelper.verifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull(sorted);
} catch (IllegalStateException e) {
  if (e.getMessage().contains("atomicUpdateGroup must be compacted together")) {
    // include the missing group members (or full replace) and retry
  } else { throw e; }
}

Prevention

Treat atomicUpdateGroups as atomic units: never filter out individual members.
Confirm segment counts per partition group against atomicUpdateGroupSize before submitting the task.
Investigate ingestion failures that publish fewer segments than the declared group size.

When it happens

Trigger: verifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull, mid-loop branch: when moving from one rootPartitionRange to the next, the running atomicUpdateGroupSize counter (segments seen in the finished group) differs from curSegment.getAtomicUpdateGroupSize(), e.g. groupSize=3 but only 2 segments supplied.

Common situations: A member segment of a multi-partition atomicUpdateGroup was deleted, tombstoned, or excluded by a segment filter; segments ingested with different partitioning (atomicUpdateGroupSize changed) coexist in one interval; manual segment selection for compaction dropped one shard.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c18115219b20a4e1. Report an issue: GitHub.