{"record":{"id":"0b11d74d17abf780","repo":"apache/iceberg","slug":"sort-order-specified-for-job-doesn-t-match-any-0b11d7","errorCode":null,"errorMessage":"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","messagePattern":"Sort order specified for job (.+?) doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java","lineNumber":126,"sourceCode":"  @Override\n  public void doRewrite(String groupId, RewriteFileGroup fileGroup) {\n    Dataset<Row> scanDF = spark().read().format(\"iceberg\").load(groupId);\n\n    Dataset<Row> sortedDF =\n        sortedDF(\n            scanDF,\n            sortFunction(\n                fileGroup.fileScanTasks(),\n                spec(fileGroup.outputSpecId()),\n                fileGroup.expectedOutputFiles()));\n\n    org.apache.iceberg.SortOrder sortOrderInJobSpec = sortOrder();\n\n    org.apache.iceberg.SortOrder maybeMatchingTableSortOrder =\n        SortOrderUtil.findTableSortOrder(table(), sortOrder());\n\n    if (sortOrderInJobSpec.isSorted() && maybeMatchingTableSortOrder.isUnsorted()) {\n      LOG.warn(\n          \"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files\",\n          Spark3Util.describe(sortOrderInJobSpec));\n    }\n\n    sortedDF\n        .write()\n        .format(\"iceberg\")\n        .option(SparkWriteOptions.TARGET_FILE_SIZE_BYTES, fileGroup.maxOutputFileSize())\n        .option(SparkWriteOptions.USE_TABLE_DISTRIBUTION_AND_ORDERING, \"false\")\n        .option(SparkWriteOptions.OUTPUT_SPEC_ID, fileGroup.outputSpecId())\n        .option(SparkWriteOptions.OUTPUT_SORT_ORDER_ID, maybeMatchingTableSortOrder.orderId())\n        .mode(\"append\")\n        .save(groupId);\n  }\n\n  private Function<Dataset<Row>, Dataset<Row>> sortFunction(\n      List<FileScanTask> group, PartitionSpec outputSpec, int expectedOutputFiles) {\n    SortOrder[] ordering = Spark3Util.toOrdering(outputSortOrder(group, outputSpec));","sourceCodeStart":108,"sourceCodeEnd":144,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java#L108-L144","documentation":"SparkShufflingFileRewriteRunner sorts rewritten data per a job-specified sort order, but rewritten data files are only marked sorted in metadata if the job's order matches a table sort order. When SortOrderUtil.findTableSortOrder finds no match, it logs this warning and the rewritten files will be written as unsorted in manifests.","triggerScenarios":"Calling doRewrite with a sort order built by the rewrite job (e.g. sort_order option) that isSorted() but does not equal any of the table's declared sort orders.","commonSituations":"Sort option in the rewrite differs from the table's ALTER TABLE ... SORT BY order (extra/missing columns, different direction or null order); table sort order changed after the rewrite job spec was created.","solutions":["Align the rewrite job's sort option with a table sort order (same columns, direction, null ordering)","Update the table's sort order to match the desired ordering, then re-run","Ignore the warning if unsorted manifest marking is acceptable; data is still physically sorted"],"exampleFix":"// before\n.option(\"sort_order\", \"id ASC\")  // table sort order is id DESC\n// after\n.option(\"sort_order\", \"id DESC\") // matches table sort order","handlingStrategy":"validation","validationCode":"org.apache.iceberg.SortOrder jobOrder = buildSortOrder(...);\nif (jobOrder.isSorted() && SortOrderUtil.findTableSortOrder(table, jobOrder).isUnsorted()) {\n  // fix option or update table sort order before rewriting\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Keep the rewrite sort option identical to a table sort order (columns, direction, null order)","Update table sort order via ALTER TABLE ... SORT BY before rewrites","Compare with SortOrderUtil.findTableSortOrder before running"],"tags":["spark","sort-order","rewrite","metadata"],"backgroundTag":"ignored-config-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}