apache/druid · error · MSQException

Unknown

Unknown

Error message

Query[%s] cannot store compaction state in segments as shard spec of unsupported type[%s].

What it means

When an MSQ query must store compaction state into output segments, ControllerImpl derives a partition spec from each segment's shard spec. Only LinearShardSpec and NumberedShardSpec are supported; encountering any other shard spec type throws MSQException with an UnknownFault, since MSQ never creates such shard specs itself.

Source

Thrown at multi-stage-query/src/main/java/org/apache/druid/msq/exec/ControllerImpl.java:1856

    final PartitionsSpec partitionSpec;
    // shardSpec is absent in the absence of segments, which happens when only tombstones are generated by an
    // MSQControllerTask.
    if (shardSpec != null) {
      if (Objects.equals(shardSpec.getType(), ShardSpec.Type.RANGE)) {
        List<String> partitionDimensions = ((DimensionRangeShardSpec) shardSpec).getDimensions();
        // Effective maxRowsPerSegment is propagated as rowsPerSegment in MSQ
        partitionSpec = new DimensionRangePartitionsSpec(
            null,
            tuningConfig.getRowsPerSegment(),
            partitionDimensions,
            false
        );
      } else if (Objects.equals(shardSpec.getType(), ShardSpec.Type.NUMBERED)) {
        // MSQ tasks don't use maxTotalRows. Hence using LONG.MAX_VALUE.
        partitionSpec = new DynamicPartitionsSpec(tuningConfig.getRowsPerSegment(), Long.MAX_VALUE);
      } else {
        // SingleDimenionShardSpec and other shard specs are never created in MSQ.
        throw new MSQException(
            UnknownFault.forMessage(
                StringUtils.format(
                    "Query[%s] cannot store compaction state in segments as shard spec of unsupported type[%s].",
                    queryDef.getQueryId(),
                    shardSpec.getType()
                )));
      }
    } else if (clusterBy != null && !clusterBy.getColumns().isEmpty()) {
      // Effective maxRowsPerSegment is propagated as rowsPerSegment in MSQ
      partitionSpec = new DimensionRangePartitionsSpec(
          null,
          tuningConfig.getRowsPerSegment(),
          clusterBy.getColumns()
                   .stream()
                   .map(KeyColumn::columnName).collect(Collectors.toList()),
          false
      );
    } else {

View on GitHub (pinned to 9b90983fd2)

Solutions

Re-generate or recompact the input segments so they use linear/numbered shard specs before running the MSQ job
Run native compaction instead of MSQ compaction for datasources containing unsupported shard specs
Avoid enabling 'store compaction state in segments' for these datasources
Upgrade Druid if a newer release adds support for the shard spec type

Defensive patterns

Strategy: try-catch

Validate before calling

for (DataSegment seg : inputSegments) { if (seg.getShardSpec().getType() != ShardSpec.Type.LINEAR && seg.getShardSpec().getType() != ShardSpec.Type.NUMBERED) throw new IllegalStateException("unsupported shard type: " + seg.getShardSpec().getType()); }

Try / catch

try { client.runMsq(compactionQuery); } catch (MSQException e) { if (e.getFault() instanceof UnknownFault && e.getMessage().contains("shard spec of unsupported type")) { recompactNativelyFirst(); } else { throw e; } }

Prevention

Check datasource segment shard specs before MSQ compaction
Recompact with linear/numbered sharding first
Avoid mixing native-compacted single-dim segments into MSQ compaction

When it happens

Trigger: Running an MSQ query (e.g. INSERT/REPLACE or compaction via MSQ) whose input/target segments carry a shard spec whose type is neither LINEAR nor NUMBERED (e.g. SingleDimensionShardSpec from native compaction) while storing compaction state in segments.

Common situations: Compacting with MSQ over segments previously produced by native compaction with hash/single-dimension sharding; mixing segment generations created by different engines.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/druid@9b90983fd2 (2026-09-07). Data as JSON: /api/errors/975026bc32bf8d20. Report an issue: GitHub.