apache/druid · error · IllegalArgumentException

Segments to drop must all be part of the same datasource

Error message

Segments to drop must all be part of the same datasource

What it means

SqlSegmentsMetadataQuery.markSegments() validates that every SegmentId in the passed collection belongs to the same datasource before building the batch SQL for mark-as-used/mark-as-unused. If any segment's datasource differs from the first element's datasource, it throws IllegalArgumentException, because the generated UPDATE statement is scoped to a single datasource.

Source

Thrown at server/src/main/java/org/apache/druid/metadata/SqlSegmentsMetadataQuery.java:785

  {
    return markSegments(segmentIds, false, updateTime);
  }

  /**
   * Marks the given segments as either used or unused.
   *
   * @return the number of segments actually modified.
   */
  private int markSegments(final Set<SegmentId> segmentIds, final boolean used, DateTime updateTime)
  {
    final String dataSource;

    if (segmentIds.isEmpty()) {
      return 0;
    } else {
      dataSource = segmentIds.iterator().next().getDataSource();
      if (segmentIds.stream().anyMatch(segment -> !dataSource.equals(segment.getDataSource()))) {
        throw new IAE("Segments to drop must all be part of the same datasource");
      }
    }

    final PreparedBatch batch =
        handle.prepareBatch(
            StringUtils.format(
                "UPDATE %s SET used = ?, used_status_last_updated = ? WHERE datasource = ? AND id = ?",
                dbTables.getSegmentsTable()
            )
        );

    for (SegmentId segmentId : segmentIds) {
      batch.add(used, updateTime.toString(), dataSource, segmentId.toString());
    }

    final int[] segmentChanges = batch.execute();
    return computeNumChangedSegments(
        segmentIds.stream().map(SegmentId::toString).collect(Collectors.toList()),

View on GitHub (pinned to 9b90983fd2)

Solutions

  1. Partition your SegmentIds by dataSource and call markSegments once per datasource.
  2. Filter the input list to a single datasource before invoking markSegmentsAsUsed/markSegmentsAsUnused.
  3. Add an assertion/grouping step in the caller (e.g., Collectors.groupingBy(SegmentId::getDataSource)).

Example fix

// before
segmentsMetadataManager.markSegmentsAsUnused(mixedSegments);
// after
mixedSegments.stream()
    .collect(Collectors.groupingBy(SegmentId::getDataSource))
    .forEach((ds, segs) -> segmentsMetadataManager.markSegmentsAsUnused(segs));
Defensive patterns

Strategy: validation

Validate before calling

Set<String> datasources = segmentIds.stream()
    .map(SegmentId::getDataSource)
    .collect(Collectors.toSet());
if (datasources.size() > 1) {
  throw new IllegalArgumentException("mixed datasources: " + datasources);
}

Type guard

boolean sameDatasource(List<SegmentId> ids) {
  return ids.isEmpty() || ids.stream().map(SegmentId::getDataSource).distinct().count() <= 1;
}

Try / catch

try {
  manager.markSegmentsAsUnused(segmentIds);
} catch (IllegalArgumentException e) {
  // group by datasource and retry per datasource
}

Prevention

When it happens

Trigger: Passing a mixed collection of SegmentIds from two or more datasources to markSegmentsAsUsed(), markSegmentsAsUnused(), markSegmentAsUsed()/markSegmentsUnused batching paths that route through markSegments().

Common situations: Bulk unused/used operations built from a segment list spanning multiple datasources; scripts or tooling that combine segment sets across datasources; a bug in calling code that forgot to group by dataSource before calling the metadata manager.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/5986ee1385a87230. Report an issue: GitHub.