{"record":{"id":"2beda41a500b5112","repo":"apache/iceberg","slug":"sort-order-specified-for-job-doesn-t-match-any-2beda4","errorCode":null,"errorMessage":"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","messagePattern":"Sort order specified for job (.+?) doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"info","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java","lineNumber":132,"sourceCode":"            .format(\"iceberg\")\n            .option(SparkReadOptions.SCAN_TASK_SET_ID, groupId)\n            .load(groupId);\n\n    Dataset<Row> sortedDF =\n        sortedDF(\n            scanDF,\n            sortFunction(\n                fileGroup.fileScanTasks(),\n                spec(fileGroup.outputSpecId()),\n                fileGroup.expectedOutputFiles()));\n\n    org.apache.iceberg.SortOrder sortOrderInJobSpec = sortOrder();\n\n    org.apache.iceberg.SortOrder maybeMatchingTableSortOrder =\n        SortOrderUtil.findTableSortOrder(table(), sortOrder());\n\n    if (sortOrderInJobSpec.isSorted() && maybeMatchingTableSortOrder.isUnsorted()) {\n      LOG.warn(\n          \"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files\",\n          Spark3Util.describe(sortOrderInJobSpec));\n    }\n\n    sortedDF\n        .write()\n        .format(\"iceberg\")\n        .option(SparkWriteOptions.REWRITTEN_FILE_SCAN_TASK_SET_ID, groupId)\n        .option(SparkWriteOptions.TARGET_FILE_SIZE_BYTES, fileGroup.maxOutputFileSize())\n        .option(SparkWriteOptions.USE_TABLE_DISTRIBUTION_AND_ORDERING, \"false\")\n        .option(SparkWriteOptions.OUTPUT_SPEC_ID, fileGroup.outputSpecId())\n        .option(SparkWriteOptions.OUTPUT_SORT_ORDER_ID, maybeMatchingTableSortOrder.orderId())\n        .mode(\"append\")\n        .save(groupId);\n  }\n\n  private Function<Dataset<Row>, Dataset<Row>> sortFunction(\n      List<FileScanTask> group, PartitionSpec outputSpec, int expectedOutputFiles) {","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java#L114-L150","documentation":"WARN from SparkShufflingFileRewriteRunner.doRewrite: the sort order requested for the rewrite job does not match any of the table's declared sort orders, so while the data IS physically sorted by the job spec, the rewritten manifest entries cannot be marked with that sort order (Iceberg only marks files with a table-declared sort order).","triggerScenarios":"Calling rewrite_data_files with sort_order/strategy options (e.g. via SparkActions...sort(asc(\"col\"))) where SortOrderUtil.findTableSortOrder returns unsorted because the job's SortOrder isn't identical to any table sort order (field ids, direction, null ordering must all match).","commonSituations":"Specifying a sort on columns not covered by the table's sorted sort order; sort order set on the table after files were written; case/transform mismatch (identity vs truncate) between job spec and table spec; forgetting to first set the table sort order via ALTER TABLE ... ORDER BY.","solutions":["Set the table sort order to match the job: ALTER TABLE t WRITE ORDERED BY col, or update sort order via UpdateSortOrder.","Change the job's sort option to exactly match an existing table sort order.","Accept the warning if physical sorting is all you need and you don't rely on sorted-manifest metadata for planning."],"exampleFix":"// before\naction.sort(asc(\"event_time\")).execute(); // table has different/null sort order\n// after: align table sort order first\ntable.updateSortOrder().addField(\"event_time\").commit();\naction.sort(asc(\"event_time\")).execute();","handlingStrategy":"validation","validationCode":"SortOrder tableOrder = table.sortOrder(); SortOrder jobOrder = /* SortOrder built from job spec */; boolean matches = SortOrderUtil.findTableSortOrder(table, jobOrder).isSorted();","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Set the table sort order (ALTER TABLE ... WRITE ORDERED BY) before sorting rewrites","Ensure job sort spec matches table order in field ids, direction, and null ordering","Don't rely on manifest sort metadata unless the warning is absent"],"tags":["logging","spark","sort-order","metadata"],"backgroundTag":"sort-order-mismatch","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}