apache/druid · error · IllegalArgumentException

Cleaned partition map

Error message

Cleaned partition map [%s] contains unexpected partition ID [%s], original partition map: [%s]

What it means

After computing metadata with expired partitions removed, the supervisor sanity-checks that the cleaned partition map is a subset of the original map. If cleaning somehow introduced a partition ID that was not there before, the invariant is broken and IllegalArgumentException is thrown naming the cleaned map, the offending partition, and the original map.

Solutions

1. Fix the custom createDataSourceMetadataWithExpiredPartitions implementation so it only removes or marks partitions from the input map, never adds new keys.
2. Log the original and cleaned maps (as the error already prints them) and diff to find the injected partition ID.
3. Verify the expiredPartitionIds set passed in contains only IDs from the original partition map.

Example fix

// before: override adds rebuilt partition groups into cleaned metadata
cleanedMap.putAll(recomputedPartitions);
// after: only retain partitions from the original map
cleanedMap.keySet().retainAll(oldPartitionSeqNos.keySet());

if (!oldPartitionSeqNos.keySet().containsAll(cleanedPartitionSeqNos.keySet())) {
    throw new IllegalStateException("expiration hook added partitions");
}

try { applyCleanedMetadata(); } catch (IAE e) { log.error("Non-subset cleaned map: {}", e.getMessage()); }

When it happens

Trigger: createDataSourceMetadataWithExpiredPartitions returns metadata whose partition map contains a key absent from the pre-cleaning map — i.e. a buggy or non-compliant override of the expiration hook.

Common situations: Custom supervisor implementations whose expiration override adds partitions instead of only removing/marking them; concurrent modification of partition maps during expiration processing.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• authResult.getErrorMessage()
• Base sequence names do not match for the tasks in the task…
• bitmapEncoding[ ] does not match indexSpec.bitmap[ ]
• Cannot copy [%,d] bytes, only have [%,d] buffered
• Cannot delete [%,d] bytes, only have [%,d] buffered

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/c2ea41e058a09351. Report an issue: GitHub.

Appendix: source

Thrown at indexing-service/src/main/java/org/apache/druid/indexing/seekablestream/supervisor/SeekableStreamSupervisor.java:3619

   * @param oldMetadata     metadata containing expired partitions.
   * @param cleanedMetadata new metadata without expired partitions, generated by the subclass
   */
  private void validateMetadataPartitionExpiration(
      Set<PartitionIdType> newlyExpiredPartitions,
      SeekableStreamDataSourceMetadata<PartitionIdType, SequenceOffsetType> oldMetadata,
      SeekableStreamDataSourceMetadata<PartitionIdType, SequenceOffsetType> cleanedMetadata
  )
  {
    Map<PartitionIdType, SequenceOffsetType> oldPartitionSeqNos = oldMetadata.getSeekableStreamSequenceNumbers()
                                                                             .getPartitionSequenceNumberMap();

    Map<PartitionIdType, SequenceOffsetType> cleanedPartitionSeqNos = cleanedMetadata.getSeekableStreamSequenceNumbers()
                                                                                     .getPartitionSequenceNumberMap();

    for (Entry<PartitionIdType, SequenceOffsetType> cleanedPartitionSeqNo : cleanedPartitionSeqNos.entrySet()) {
      if (!oldPartitionSeqNos.containsKey(cleanedPartitionSeqNo.getKey())) {
        // cleaning the expired partitions added a partition somehow
        throw new IAE(
            "Cleaned partition map [%s] contains unexpected partition ID [%s], original partition map: [%s]",
            cleanedPartitionSeqNos,
            cleanedPartitionSeqNo.getKey(),
            oldPartitionSeqNos
        );
      }

      SequenceOffsetType oldOffset = oldPartitionSeqNos.get(cleanedPartitionSeqNo.getKey());
      if (newlyExpiredPartitions.contains(cleanedPartitionSeqNo.getKey())) {
        // this is a newly expired partition, check that we did actually mark it as expired
        if (!isShardExpirationMarker(cleanedPartitionSeqNo.getValue())) {
          throw new IAE(
              "Newly expired partition [%] was not marked as expired in the cleaned partition map [%s], original partition map: [%s]",
              cleanedPartitionSeqNo.getKey(),
              cleanedPartitionSeqNos,
              oldPartitionSeqNos
          );
        }

View on GitHub (pinned to 9b90983fd2)