apache/druid · warning

Mismatched kafka and task partitions: Missing Task…

Error message

Mismatched kafka and task partitions: Missing Task Partitions %s, Missing Kafka Partitions %s

What it means

KafkaSupervisor.getPartitionRecordLag() logs a warning when the set of partitions the Kafka stream reports differs from the set of partitions currently assigned to tasks. It compares latestSequencesFromStream keys against highestIngestedOffsets keys and dumps the missing sets. Lag values cannot be computed per-partition when the sets disagree.

Solutions

1. Wait for the supervisor to converge; this is often transient during partition reassignment.
2. If partitions were added, verify taskCount/multiStageEngagement and let the supervisor spawn tasks for new partitions.
3. Never decrease a Kafka topic's partition count; if it happened, restore it or rebuild the datasource.
4. Check supervisor task state via /druid/indexer/supervisor status endpoints for stuck assignments.

Set<Long> kafka = latestSequencesFromStream.keySet();
Set<Long> task = highestIngestedOffsets.keySet();
if (!kafka.equals(task)) { /* skip lag computation or alert on persistent mismatch */ }

When it happens

Trigger: Calling getPartitionRecordLag() (via partitionRecordLag) when Kafka has partitions with no active task, or tasks track partitions that no longer exist in the topic (partition count changed or task lag).

Common situations: Topic partition count increased but tasks not yet reassigned; partition count decreased (unsupported shrink); supervisor still starting up or tasks in unstable state; monitoring scrape during reassignment.

Related errors

• Could not fetch partitions for topic/stream
• got null sequence number for partition
• Killing task[ ] as it failed to return start time.
• No such previous checkpoint
• No valid task counts after applying constraints for…

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/a9e83d9281529012. Report an issue: GitHub.

Appendix: source

Thrown at extensions-core/kafka-indexing-service/src/main/java/org/apache/druid/indexing/kafka/supervisor/KafkaSupervisor.java:285

    return taskList;
  }

  @Override
  protected Map<KafkaTopicPartition, Long> getPartitionRecordLag()
  {
    OffsetSnapshot<KafkaTopicPartition, Long> offsetSnapshot = offsetSnapshotRef.get();
    Map<KafkaTopicPartition, Long> latestSequencesFromStream = offsetSnapshot.getLatestOffsetsFromStream();
    Map<KafkaTopicPartition, Long> highestIngestedOffsets = offsetSnapshot.getHighestIngestedOffsets();

    if (latestSequencesFromStream.isEmpty()) {
      return null;
    }

    Set<KafkaTopicPartition> kafkaPartitions = latestSequencesFromStream.keySet();
    Set<KafkaTopicPartition> taskPartitions = highestIngestedOffsets.keySet();
    if (!kafkaPartitions.equals(taskPartitions)) {
      try {
        log.warn("Mismatched kafka and task partitions: Missing Task Partitions %s, Missing Kafka Partitions %s",
                sortingMapper.writeValueAsString(Sets.difference(kafkaPartitions, taskPartitions)),
                 sortingMapper.writeValueAsString(Sets.difference(taskPartitions, kafkaPartitions)));
      }
      catch (JsonProcessingException e) {
        throw DruidException.defensive("Failed to serialize KafkaTopicPartition when getting partition record lag: %s",
                                       e.getMessage());
      }
    }

    return getRecordLagPerPartitionInLatestSequences(offsetSnapshot);
  }

  @Nullable
  @Override
  protected Map<KafkaTopicPartition, Long> getPartitionTimeLag()
  {
    return partitionToTimeLag;
  }

View on GitHub (pinned to 9b90983fd2)