{"record":{"id":"034af947354e6e46","repo":"apache/iceberg","slug":"sort-order-specified-for-job-doesn-t-match-any","errorCode":null,"errorMessage":"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","messagePattern":"Sort order specified for job (.+?) doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java","lineNumber":132,"sourceCode":"            .format(\"iceberg\")\n            .option(SparkReadOptions.SCAN_TASK_SET_ID, groupId)\n            .load(groupId);\n\n    Dataset<Row> sortedDF =\n        sortedDF(\n            scanDF,\n            sortFunction(\n                fileGroup.fileScanTasks(),\n                spec(fileGroup.outputSpecId()),\n                fileGroup.expectedOutputFiles()));\n\n    org.apache.iceberg.SortOrder sortOrderInJobSpec = sortOrder();\n\n    org.apache.iceberg.SortOrder maybeMatchingTableSortOrder =\n        SortOrderUtil.findTableSortOrder(table(), sortOrder());\n\n    if (sortOrderInJobSpec.isSorted() && maybeMatchingTableSortOrder.isUnsorted()) {\n      LOG.warn(\n          \"Sort order specified for job {} doesn't match any table sort orders, rewritten files will not be marked as sorted in the manifest files\",\n          Spark3Util.describe(sortOrderInJobSpec));\n    }\n\n    sortedDF\n        .write()\n        .format(\"iceberg\")\n        .option(SparkWriteOptions.REWRITTEN_FILE_SCAN_TASK_SET_ID, groupId)\n        .option(SparkWriteOptions.TARGET_FILE_SIZE_BYTES, fileGroup.maxOutputFileSize())\n        .option(SparkWriteOptions.USE_TABLE_DISTRIBUTION_AND_ORDERING, \"false\")\n        .option(SparkWriteOptions.OUTPUT_SPEC_ID, fileGroup.outputSpecId())\n        .option(SparkWriteOptions.OUTPUT_SORT_ORDER_ID, maybeMatchingTableSortOrder.orderId())\n        .mode(\"append\")\n        .save(groupId);\n  }\n\n  private Function<Dataset<Row>, Dataset<Row>> sortFunction(\n      List<FileScanTask> group, PartitionSpec outputSpec, int expectedOutputFiles) {","sourceCodeStart":114,"sourceCodeEnd":150,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/actions/SparkShufflingFileRewriteRunner.java#L114-L150","documentation":"SparkShufflingFileRewriteRunner sorts data according to the sort order given in the rewrite job spec. If that job sort order doesn't match any of the table's declared sort orders, SortOrderUtil.findTableSortOrder yields unsorted, so rewritten files cannot be flagged as sorted in their manifests; this warning explains that consequence.","triggerScenarios":"Running rewrite_data_files with a sort_strategy/sort order (e.g. custom sort columns via procedure options) whose org.apache.iceberg.SortOrder is sorted but does not equal any sortOrder declared in table metadata, so SortOrderUtil.findTableSortOrder returns unsorted.","commonSituations":"Sort columns/transform/null-order in the job spec differing from the table's REPLACE/ALTER ... SORT BY order; table sort order changed after the job spec was written; using z-order strategy (job order never matches a table sort order by design).","solutions":["Align the job's sort spec with the table's declared sort order (same columns, transforms, directions, null ordering), or update the table's sort order to match.","Run ALTER TABLE ... WRITE ORDERED BY to set a table sort order that the rewrite can match.","Accept the warning if files don't need sort-order metadata (data is still physically sorted; only manifest annotation is skipped).","Verify the job's describe(sortOrder) output in the log against the table's current sort order via DESCRIBE TABLE / metadata."],"exampleFix":"// before: job sort doesn't match table order, files unmarked as sorted\nspark.sql(\"CALL cat.sys.rewrite_data_files(table => 'db.t', \" +\n  \"strategy => 'sort', where => 'id > 0', \" +\n  \"sort_order => 'id')\");\n// after: make table sort order match the job spec first\nspark.sql(\"ALTER TABLE db.t WRITE ORDERED BY id\");\nspark.sql(\"CALL cat.sys.rewrite_data_files(table => 'db.t', \" +\n  \"strategy => 'sort', sort_order => 'id')\");","handlingStrategy":"validation","validationCode":"SortOrder jobOrder = sortOrder();\nSortOrder tableOrder = SortOrderUtil.findTableSortOrder(table(), jobOrder);\nif (jobOrder.isSorted() && tableOrder.isUnsorted()) {\n  // fix job spec or ALTER TABLE WRITE ORDERED BY before rewriting\n}","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Declare the table's sort order (ALTER TABLE ... WRITE ORDERED BY) matching your rewrite job spec.","Include transforms, direction, and null-order in comparisons — any mismatch invalidates the match.","After changing table sort order, update rewrite job configs.","Check the log's describe(sortOrder) output against table metadata before large rewrites."],"tags":["spark","sort-order","rewrite-data-files","metadata"],"backgroundTag":"invalid-argument-value","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}