apache/iceberg · warning

Sort order specified for job {} doesn't match any table sort

Error message

Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files

What it means

SparkShufflingFileRewriteRunner sorts rewritten data per a job-specified sort order, but rewritten data files are only marked sorted in metadata if the job's order matches a table sort order. When SortOrderUtil.findTableSortOrder finds no match, it logs this warning and the rewritten files will be written as unsorted in manifests.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java:126

  @Override
  public void doRewrite(String groupId, RewriteFileGroup fileGroup) {
    Dataset<Row> scanDF = spark().read().format("iceberg").load(groupId);

    Dataset<Row> sortedDF =
        sortedDF(
            scanDF,
            sortFunction(
                fileGroup.fileScanTasks(),
                spec(fileGroup.outputSpecId()),
                fileGroup.expectedOutputFiles()));

    org.apache.iceberg.SortOrder sortOrderInJobSpec = sortOrder();

    org.apache.iceberg.SortOrder maybeMatchingTableSortOrder =
        SortOrderUtil.findTableSortOrder(table(), sortOrder());

    if (sortOrderInJobSpec.isSorted() && maybeMatchingTableSortOrder.isUnsorted()) {
      LOG.warn(
          "Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files",
          Spark3Util.describe(sortOrderInJobSpec));
    }

    sortedDF
        .write()
        .format("iceberg")
        .option(SparkWriteOptions.TARGET_FILE_SIZE_BYTES, fileGroup.maxOutputFileSize())
        .option(SparkWriteOptions.USE_TABLE_DISTRIBUTION_AND_ORDERING, "false")
        .option(SparkWriteOptions.OUTPUT_SPEC_ID, fileGroup.outputSpecId())
        .option(SparkWriteOptions.OUTPUT_SORT_ORDER_ID, maybeMatchingTableSortOrder.orderId())
        .mode("append")
        .save(groupId);
  }

  private Function<Dataset<Row>, Dataset<Row>> sortFunction(
      List<FileScanTask> group, PartitionSpec outputSpec, int expectedOutputFiles) {
    SortOrder[] ordering = Spark3Util.toOrdering(outputSortOrder(group, outputSpec));

View on GitHub (pinned to 86d9c8fc54)

Solutions

  1. Align the rewrite job's sort option with a table sort order (same columns, direction, null ordering)
  2. Update the table's sort order to match the desired ordering, then re-run
  3. Ignore the warning if unsorted manifest marking is acceptable; data is still physically sorted

Example fix

// before
.option("sort_order", "id ASC")  // table sort order is id DESC
// after
.option("sort_order", "id DESC") // matches table sort order
Defensive patterns

Strategy: validation

Validate before calling

org.apache.iceberg.SortOrder jobOrder = buildSortOrder(...);
if (jobOrder.isSorted() && SortOrderUtil.findTableSortOrder(table, jobOrder).isUnsorted()) {
  // fix option or update table sort order before rewriting
}

Prevention

When it happens

Trigger: Calling doRewrite with a sort order built by the rewrite job (e.g. sort_order option) that isSorted() but does not equal any of the table's declared sort orders.

Common situations: Sort option in the rewrite differs from the table's ALTER TABLE ... SORT BY order (extra/missing columns, different direction or null order); table sort order changed after the rewrite job spec was created.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0b11d74d17abf780. Report an issue: GitHub.