apache/iceberg · warning

Sort order specified for job {} doesn't match any table sort

Error message

Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files

What it means

A LOG.warn in SparkShufflingFileRewriteRunner.doRewrite: the sort order specified for the rewrite job does not match any sort order defined on the table (via SortOrderUtil.findTableSortOrder), so the rewritten data files cannot be recorded as sorted in their manifests. Data is still written sorted, but metadata will show the files as unsorted, weakening future read planning.

Source

Thrown at spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java:126

  @Override
  public void doRewrite(String groupId, RewriteFileGroup fileGroup) {
    Dataset<Row> scanDF = spark().read().format("iceberg").load(groupId);

    Dataset<Row> sortedDF =
        sortedDF(
            scanDF,
            sortFunction(
                fileGroup.fileScanTasks(),
                spec(fileGroup.outputSpecId()),
                fileGroup.expectedOutputFiles()));

    org.apache.iceberg.SortOrder sortOrderInJobSpec = sortOrder();

    org.apache.iceberg.SortOrder maybeMatchingTableSortOrder =
        SortOrderUtil.findTableSortOrder(table(), sortOrder());

    if (sortOrderInJobSpec.isSorted() && maybeMatchingTableSortOrder.isUnsorted()) {
      LOG.warn(
          "Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files",
          Spark3Util.describe(sortOrderInJobSpec));
    }

    sortedDF
        .write()
        .format("iceberg")
        .option(SparkWriteOptions.TARGET_FILE_SIZE_BYTES, fileGroup.maxOutputFileSize())
        .option(SparkWriteOptions.USE_TABLE_DISTRIBUTION_AND_ORDERING, "false")
        .option(SparkWriteOptions.OUTPUT_SPEC_ID, fileGroup.outputSpecId())
        .option(SparkWriteOptions.OUTPUT_SORT_ORDER_ID, maybeMatchingTableSortOrder.orderId())
        .mode("append")
        .save(groupId);
  }

  private Function<Dataset<Row>, Dataset<Row>> sortFunction(
      List<FileScanTask> group, PartitionSpec outputSpec, int expectedOutputFiles) {
    SortOrder[] ordering = Spark3Util.toOrdering(outputSortOrder(group, outputSpec));

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Align the job's sort order with the table's declared sort order (use the same WRITE ORDERED BY definition), so rewritten files are marked sorted in manifests.
2. If the new sort is intentional, first update the table sort order: ALTER TABLE db.t WRITE ORDERED BY ... then run the rewrite.
3. Ignore the warning if you do not care about sorted-metadata benefit, but expect readers not to leverage sort-based pruning for these files.

Example fix

// before
CALL catalog.system.rewrite_data_files(table => 'db.t', strategy => 'sort', sort_order => 'id ASC NULLS FIRST')
-- table is WRITE ORDERED BY id ASC NULLS LAST => mismatch
// after
ALTER TABLE db.t WRITE ORDERED BY id ASC NULLS LAST;
CALL catalog.system.rewrite_data_files(table => 'db.t', strategy => 'sort', sort_order => 'id ASC NULLS LAST')

org.apache.iceberg.SortOrder jobOrder = sortOrder();
org.apache.iceberg.SortOrder tableOrder = SortOrderUtil.findTableSortOrder(table(), jobOrder);
if (jobOrder.isSorted() && tableOrder.isUnsorted()) {
  // align job sort order with a declared table sort order, or update the table first
  throw new IllegalArgumentException("Job sort order does not match any table sort order");
}

When it happens

Trigger: Running rewriteDataFiles with strategy 'sort' and a sort_order/sort expression that differs from all table sort orders defined via ALTER TABLE ... WRITE ORDERED BY, then executing doRewrite.

Common situations: Ad-hoc compaction with a custom sort that diverges from the declared table sort order; table sort order changed after the rewrite job config was written; using SQL-defined sort expressions whose identity (nulls ordering, direction) differs subtly from the table's declared order.

Related errors

• Sort order specified for job {} doesn't match any table sort
• Cannot mix identity sort columns and a Zorder sort expressio
• Sort order specified for job {} doesn't match any table sort
• Sort order specified for job {} doesn't match any table sort
• Unexpected null order: ${nullOrderingAsString}

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/d427286076288f3e. Report an issue: GitHub.