{"record":{"id":"3f071e2ce638d0c4","repo":"apache/iceberg","slug":"file-s-is-referenced-in-the-logs-of-delta-lake-ta","errorCode":null,"errorMessage":"File %s is referenced in the logs of Delta Lake table at %s, but cannot be found in the storage","messagePattern":"File (.+?) is referenced in the logs of Delta Lake table at (.+?), but cannot be found in the storage","errorType":"exception","errorClass":"NotFoundException","httpStatus":null,"severity":"error","filePath":"delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java","lineNumber":369,"sourceCode":"      RemoveFile removeFile = (RemoveFile) action;\n      path = removeFile.getPath();\n      nullableFileSize = removeFile.getSize().orElse(null);\n      partitionValues = removeFile.getPartitionValues();\n    } else {\n      throw new ValidationException(\n          \"Unexpected action type for Delta Lake: %s\", action.getClass().getSimpleName());\n    }\n\n    String fullFilePath = getFullFilePath(path, deltaLog.getPath().toString());\n    // For unpartitioned table, the partitionValues should be an empty map rather than null\n    Preconditions.checkArgument(\n        partitionValues != null,\n        String.format(\"File %s does not specify a partitionValues\", fullFilePath));\n\n    FileFormat format = determineFileFormatFromPath(fullFilePath);\n    InputFile file = deltaLakeFileIO.newInputFile(fullFilePath);\n    if (!file.exists()) {\n      throw new NotFoundException(\n          \"File %s is referenced in the logs of Delta Lake table at %s, but cannot be found in the storage\",\n          fullFilePath, deltaTableLocation);\n    }\n\n    // If the file size is not specified, the size should be read from the file\n    if (nullableFileSize != null) {\n      fileSize = nullableFileSize;\n    } else {\n      fileSize = file.getLength();\n    }\n\n    // get metrics from the file\n    MetricsConfig metricsConfig = MetricsConfig.forTable(table);\n    String nameMappingString = table.properties().get(TableProperties.DEFAULT_NAME_MAPPING);\n    NameMapping nameMapping =\n        nameMappingString != null ? NameMappingParser.fromJson(nameMappingString) : null;\n    Metrics metrics = getMetricsForFile(file, format, metricsConfig, nameMapping);\n","sourceCodeStart":351,"sourceCodeEnd":387,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/delta-lake/src/main/java/org/apache/iceberg/delta/BaseSnapshotDeltaLakeTableAction.java#L351-L387","documentation":"Thrown when a Delta log action references a data file (parquet) that no longer exists in the table's storage. The action builds an InputFile for the full path and checks existence before reading metrics; a missing file makes the migration abort since the Iceberg snapshot would reference a dangling file.","triggerScenarios":"Running SnapshotDeltaLakeTable on a Delta table whose _delta_log still contains AddFile entries for files that were deleted out-of-band (manual deletion, lifecycle policies, vacuum removing active files, wrong table location configured so paths resolve outside the real data).","commonSituations":"Deleting parquet files from S3/HDFS/ADLS manually or via bucket lifecycle rules without running Delta VACUUM; misconfigured deltaTableLocation/deltaLog path prefix so getFullFilePath points to a wrong prefix; copying _delta_log without its data files to a new location.","solutions":["Verify the file truly is absent at fullFilePath in the object store; fix the table location/IO prefix if the path is just wrong.","Restore the missing files (versioning/undelete on S3, backup) or restore the table to a snapshot whose files exist.","Run Delta Lake VACUUM with correct retention on a healthy copy and snapshot a version where all referenced files exist; never delete log-referenced files manually.","If files are intentionally gone, rewrite the Delta log (e.g. checkpoint/REORG or re-create table) so the log only references existing files, then re-run the migration."],"exampleFix":"// before: snapshot points at a moved table location\nDeltaLakeCatalogOptions.WAREHOUSE_LOCATION, \"s3://old-bucket/table\"\n// after: point the action at the location where data files actually live\nDeltaLakeCatalogOptions.WAREHOUSE_LOCATION, \"s3://current-bucket/table\"","handlingStrategy":"validation","validationCode":"InputFile f = fileIO.newInputFile(fullPath);\nif (!f.exists()) { throw new IllegalStateException(\"Missing file before snapshot: \" + fullPath); }","typeGuard":"java.util.function.Predicate<String> fileExists = p -> fileIO.newInputFile(p).exists();","tryCatchPattern":"try {\n  snapshotTable.execute();\n} catch (NotFoundException e) {\n  // log fullFilePath from message, verify storage location/restore file\n}","preventionTips":["Never delete Delta data files without VACUUM; exclude active-file paths from lifecycle policies.","Verify the table location prefix matches where data files actually live.","Run a pre-flight existence check of AddFile paths before snapshotting."],"tags":["delta-lake","storage","missing-file"],"backgroundTag":"file-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}