{"record":{"id":"d427286076288f3e","repo":"apache/iceberg","slug":"sort-order-specified-for-job-doesn-t-match-any-d42728","errorCode":null,"errorMessage":"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","messagePattern":"Sort order specified for job (.+?) doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java","lineNumber":126,"sourceCode":"  @Override\n  public void doRewrite(String groupId, RewriteFileGroup fileGroup) {\n    Dataset<Row> scanDF = spark().read().format(\"iceberg\").load(groupId);\n\n    Dataset<Row> sortedDF =\n        sortedDF(\n            scanDF,\n            sortFunction(\n                fileGroup.fileScanTasks(),\n                spec(fileGroup.outputSpecId()),\n                fileGroup.expectedOutputFiles()));\n\n    org.apache.iceberg.SortOrder sortOrderInJobSpec = sortOrder();\n\n    org.apache.iceberg.SortOrder maybeMatchingTableSortOrder =\n        SortOrderUtil.findTableSortOrder(table(), sortOrder());\n\n    if (sortOrderInJobSpec.isSorted() && maybeMatchingTableSortOrder.isUnsorted()) {\n      LOG.warn(\n          \"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files\",\n          Spark3Util.describe(sortOrderInJobSpec));\n    }\n\n    sortedDF\n        .write()\n        .format(\"iceberg\")\n        .option(SparkWriteOptions.TARGET_FILE_SIZE_BYTES, fileGroup.maxOutputFileSize())\n        .option(SparkWriteOptions.USE_TABLE_DISTRIBUTION_AND_ORDERING, \"false\")\n        .option(SparkWriteOptions.OUTPUT_SPEC_ID, fileGroup.outputSpecId())\n        .option(SparkWriteOptions.OUTPUT_SORT_ORDER_ID, maybeMatchingTableSortOrder.orderId())\n        .mode(\"append\")\n        .save(groupId);\n  }\n\n  private Function<Dataset<Row>, Dataset<Row>> sortFunction(\n      List<FileScanTask> group, PartitionSpec outputSpec, int expectedOutputFiles) {\n    SortOrder[] ordering = Spark3Util.toOrdering(outputSortOrder(group, outputSpec));","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.2/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java#L108-L144","documentation":"A LOG.warn in SparkShufflingFileRewriteRunner.doRewrite: the sort order specified for the rewrite job does not match any sort order defined on the table (via SortOrderUtil.findTableSortOrder), so the rewritten data files cannot be recorded as sorted in their manifests. Data is still written sorted, but metadata will show the files as unsorted, weakening future read planning.","triggerScenarios":"Running rewriteDataFiles with strategy 'sort' and a sort_order/sort expression that differs from all table sort orders defined via ALTER TABLE ... WRITE ORDERED BY, then executing doRewrite.","commonSituations":"Ad-hoc compaction with a custom sort that diverges from the declared table sort order; table sort order changed after the rewrite job config was written; using SQL-defined sort expressions whose identity (nulls ordering, direction) differs subtly from the table's declared order.","solutions":["Align the job's sort order with the table's declared sort order (use the same WRITE ORDERED BY definition), so rewritten files are marked sorted in manifests.","If the new sort is intentional, first update the table sort order: ALTER TABLE db.t WRITE ORDERED BY ... then run the rewrite.","Ignore the warning if you do not care about sorted-metadata benefit, but expect readers not to leverage sort-based pruning for these files."],"exampleFix":"// before\nCALL catalog.system.rewrite_data_files(table => 'db.t', strategy => 'sort', sort_order => 'id ASC NULLS FIRST')\n-- table is WRITE ORDERED BY id ASC NULLS LAST => mismatch\n// after\nALTER TABLE db.t WRITE ORDERED BY id ASC NULLS LAST;\nCALL catalog.system.rewrite_data_files(table => 'db.t', strategy => 'sort', sort_order => 'id ASC NULLS LAST')","handlingStrategy":"validation","validationCode":"org.apache.iceberg.SortOrder jobOrder = sortOrder();\norg.apache.iceberg.SortOrder tableOrder = SortOrderUtil.findTableSortOrder(table(), jobOrder);\nif (jobOrder.isSorted() && tableOrder.isUnsorted()) {\n  // align job sort order with a declared table sort order, or update the table first\n  throw new IllegalArgumentException(\"Job sort order does not match any table sort order\");\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Declare the table sort order with WRITE ORDERED BY before sorted rewrites","Reuse the table's declared sort order in rewrite jobs instead of ad-hoc expressions","Update table sort order first when intentionally changing sort strategy","Compare nulls-first/last and direction details, not just column names"],"tags":["spark","sort-order","rewrite-data-files","metadata"],"backgroundTag":"sort-order-mismatch","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}