{"record":{"id":"37d83a8d12cd0fdc","repo":"prestodb/presto","slug":"iceberg-invalid-metadata-37d83a","errorCode":"ICEBERG_INVALID_METADATA","errorMessage":"Unable to find sort field source column in the table schema: ","messagePattern":"Unable to find sort field source column in the table schema: ","errorType":"error_code","errorClass":"PrestoException","httpStatus":null,"severity":"error","filePath":"presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergPageSink.java","lineNumber":194,"sourceCode":"                .map(IcebergColumnHandle::getType)\n                .collect(toImmutableList());\n        this.sortParameters = sortParameters;\n        if (!sortOrder.isEmpty()) {\n            ImmutableList.Builder<Integer> sortColumnIndexes = ImmutableList.builder();\n            ImmutableList.Builder<SortOrder> sortOrders = ImmutableList.builder();\n            for (SortField sortField : sortOrder) {\n                if (sortField.getSourceColumnId() == Z_ORDER.getId()) {\n                    sortColumnIndexes.add(outputSchema.columns().size());\n                    sortOrders.add(sortField.getSortOrder());\n                    types = ImmutableList.<Type>builder()\n                            .addAll(types)\n                            .add(VARBINARY)\n                            .build();\n                }\n                else {\n                    Types.NestedField column = outputSchema.findField(sortField.getSourceColumnId());\n                    if (column == null) {\n                        throw new PrestoException(ICEBERG_INVALID_METADATA, \"Unable to find sort field source column in the table schema: \" + sortField);\n                    }\n                    sortColumnIndexes.add(outputSchema.columns().indexOf(column));\n                    sortOrders.add(sortField.getSortOrder());\n                }\n            }\n            this.sortColumnIndexes = sortColumnIndexes.build();\n            this.sortOrders = sortOrders.build();\n        }\n        else {\n            this.sortColumnIndexes = ImmutableList.of();\n            this.sortOrders = ImmutableList.of();\n        }\n        this.columnTypes = types;\n    }\n\n    @Override\n    public long getCompletedBytes()\n    {","sourceCodeStart":176,"sourceCodeEnd":212,"githubUrl":"https://github.com/prestodb/presto/blob/55bb57d202de3b926896fa966c2c4a44c779634e/presto-iceberg/src/main/java/com/facebook/presto/iceberg/IcebergPageSink.java#L176-L212","documentation":"During IcebergPageSink construction, each sort field from the table's Iceberg sort order is resolved against the sink's output schema via findField(sourceColumnId). If the schema has no field with that column ID, the table metadata is internally inconsistent (sort order references a column that no longer exists), so Presto throws ICEBERG_INVALID_METADATA rather than writing corrupt data.","triggerScenarios":"Writing (INSERT/CTAS/refresh) into an Iceberg table whose sort order metadata references a source column ID absent from the table's current schema — typically after schema evolution dropped/renamed the sort column while stale sort-order entries remained, or reading a table written by another engine with inconsistent metadata.","commonSituations":"ALTER TABLE DROP/RENAME of a column that was part of the table's sort order without updating sort order; old writer versions reading tables modified by newer tooling; corrupted metadata.json hand-edited or produced by a buggy catalog migration.","solutions":["Fix the table metadata: remove or rewrite the sort order so it only references existing schema columns (e.g. via Iceberg API removeSortOrder()/replaceSortOrder or a repair tool).","Restore the dropped/renamed column, or re-add it under a new name and update the sort order to point at the current field IDs.","Roll back to a prior metadata.json (rollback_to_snapshot / register previous metadata version) where schema and sort order were consistent, then evolve the schema correctly.","Upgrade Presto/Iceberg connector versions — newer writers handle schema-evolved sort orders more gracefully."],"exampleFix":"// before: sort order references dropped column\n// schema: [1:id, 2:name], sort order: [field 3:ts DESC]  <- field 3 gone\n// after (Iceberg API repair)\ntable.replaceSortOrder().asc(\"id\").commit();  // sort only on existing fields","handlingStrategy":"validation","validationCode":"// check sort order vs current schema before writing (Iceberg API)\nSchema schema = table.schema();\nboolean consistent = table.sortOrder().fields().stream()\n    .allMatch(f -> schema.findField(f.getSourceColumnId()) != null);\nif (!consistent) throw new IllegalStateException(\"Sort order references missing columns; repair metadata before INSERT\");","typeGuard":null,"tryCatchPattern":"try { table.newAppend().appendFile(dataFile).commit(); }\ncatch (PrestoException e) {\n  if (e.getCode().equals(ICEBERG_INVALID_METADATA)) {\n    table.replaceSortOrder().commit(); // clear stale sort order, retry\n  } else throw e;\n}","preventionTips":["When dropping/renaming columns, always update (or clear) the table sort order in the same evolution operation.","Run metadata consistency checks after cross-engine writes or catalog migrations.","Avoid hand-editing metadata.json files.","Pin a recent Presto/Iceberg writer version that tolerates evolved sort orders."],"tags":["iceberg","schema","sort-order","invalid-metadata"],"backgroundTag":"schema-column-not-found","analyzedSha":"55bb57d202de3b926896fa966c2c4a44c779634e","analyzedAt":"2026-09-04T12:50:26.162Z","contentChangedAt":"2026-09-04T12:50:26.162Z","schemaVersion":2},"datasetVersion":"2026-09-11T21:17:09.523Z"}