{"record":{"id":"c18115219b20a4e1","repo":"apache/druid","slug":"all-atomicupdategroup-must-be-compacted-together","errorCode":null,"errorMessage":"All atomicUpdateGroup must be compacted together. Expected size[%s] but current size[%s]","messagePattern":"All atomicUpdateGroup must be compacted together\\. Expected size\\[(.+?)\\] but current size\\[(.+?)\\]","errorType":"exception","errorClass":"ISE","httpStatus":null,"severity":"error","filePath":"indexing-service/src/main/java/org/apache/druid/indexing/common/task/TaskLockHelper.java","lineNumber":296,"sourceCode":"        if (curSegment.getMinorVersion() != nextSegment.getMinorVersion()\n            || curSegment.getAtomicUpdateGroupSize() != nextSegment.getAtomicUpdateGroupSize()) {\n          throw new ISE(\n              \"segment[%s] and segment[%s] have the same rootPartitionRange, but different minorVersion or atomicUpdateGroupSize\",\n              curSegment,\n              nextSegment\n          );\n        }\n        atomicUpdateGroupSize++;\n      } else {\n        if (curSegment.getEndRootPartitionId() != nextSegment.getStartRootPartitionId()) {\n          throw new ISE(\n              \"Can't compact segments of non-consecutive rootPartition range. Missing partitionIds between [%s] and [%s]\",\n              curSegment.getEndRootPartitionId(),\n              nextSegment.getStartRootPartitionId()\n          );\n        }\n        if (atomicUpdateGroupSize != curSegment.getAtomicUpdateGroupSize()) {\n          throw new ISE(\n              \"All atomicUpdateGroup must be compacted together. Expected size[%s] but current size[%s]\",\n              curSegment.getAtomicUpdateGroupSize(),\n              atomicUpdateGroupSize\n          );\n        }\n        atomicUpdateGroupSize = 1;\n      }\n    }\n    if (atomicUpdateGroupSize != sortedSegments.get(sortedSegments.size() - 1).getAtomicUpdateGroupSize()) {\n      throw new ISE(\n          \"All atomicUpdateGroup must be compacted together. Expected size[%s] but current size[%s]\",\n          sortedSegments.get(sortedSegments.size() - 1).getAtomicUpdateGroupSize(),\n          atomicUpdateGroupSize\n      );\n    }\n  }\n}\n","sourceCodeStart":278,"sourceCodeEnd":314,"githubUrl":"https://github.com/apache/druid/blob/9b90983fd291f26935af934383ce360473179e4d/indexing-service/src/main/java/org/apache/druid/indexing/common/task/TaskLockHelper.java#L278-L314","documentation":"Thrown when a completed atomicUpdateGroup (a group of consecutive segments sharing one rootPartitionRange) is closed at a range boundary but the number of segments accumulated for that group does not equal the group's declared atomicUpdateGroupSize. Every atomicUpdateGroup must be fully present in the input, because partial replacement of a group cannot produce output that safely overshadows the existing segments.","triggerScenarios":"verifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull, mid-loop branch: when moving from one rootPartitionRange to the next, the running atomicUpdateGroupSize counter (segments seen in the finished group) differs from curSegment.getAtomicUpdateGroupSize(), e.g. groupSize=3 but only 2 segments supplied.","commonSituations":"A member segment of a multi-partition atomicUpdateGroup was deleted, tombstoned, or excluded by a segment filter; segments ingested with different partitioning (atomicUpdateGroupSize changed) coexist in one interval; manual segment selection for compaction dropped one shard.","solutions":["Ensure the input includes all segments of each atomicUpdateGroup (match count to the declared atomicUpdateGroupSize).","Find the missing segment(s) via the segments table (same interval and rootPartitionRange) and include or restore them.","If the group cannot be completed, re-run a full replace of the interval to regenerate a consistent set of segments.","Avoid editing/excluding individual segments of a multi-segment atomicUpdateGroup; operate at the whole-group or whole-interval level."],"exampleFix":"// before: one shard of a size-3 group filtered out\nList<DataSegment> input = all.stream().filter(s -> s.getShardSpec().getPartitionNum() != 2).collect(toList());\nverifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull(input); // ISE: expected 3, got 2\n\n// after: keep the whole group\nverifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull(all);","handlingStrategy":"validation","validationCode":"// Java: verify every atomicUpdateGroup is fully present before compaction\nMap<Integer, List<DataSegment>> byStart = segments.stream()\n    .collect(Collectors.groupingBy(DataSegment::getStartRootPartitionId));\nbyStart.forEach((start, group) -> {\n  int declared = group.get(0).getAtomicUpdateGroupSize();\n  if (group.size() != declared) {\n    throw new IllegalStateException(\"Incomplete atomicUpdateGroup at rootPartition \" + start\n        + \": expected \" + declared + \", found \" + group.size());\n  }\n});","typeGuard":null,"tryCatchPattern":"try {\n  TaskLockHelper.verifyRootPartitionIsAdjacentAndAtomicUpdateGroupIsFull(sorted);\n} catch (IllegalStateException e) {\n  if (e.getMessage().contains(\"atomicUpdateGroup must be compacted together\")) {\n    // include the missing group members (or full replace) and retry\n  } else { throw e; }\n}","preventionTips":["Treat atomicUpdateGroups as atomic units: never filter out individual members.","Confirm segment counts per partition group against atomicUpdateGroupSize before submitting the task.","Investigate ingestion failures that publish fewer segments than the declared group size."],"tags":["druid","atomic-update-group","compaction","segment-lock"],"backgroundTag":"internal-invariant-violation","analyzedSha":"9b90983fd291f26935af934383ce360473179e4d","analyzedAt":"2026-09-07T13:32:30.957Z","contentChangedAt":"2026-09-07T13:32:30.957Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}