{"record":{"id":"03bec785285f3d84","repo":"apache/iceberg","slug":"ignoring-filenotfoundexception-when-listing-partit","errorCode":null,"errorMessage":"Ignoring FileNotFoundException when listing partition of {}","messagePattern":"Ignoring FileNotFoundException when listing partition of (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java","lineNumber":312,"sourceCode":"      PartitionSpec spec,\n      SerializableConfiguration conf,\n      MetricsConfig metricsConfig,\n      NameMapping mapping,\n      boolean ignoreMissingFiles,\n      ExecutorService service) {\n    try {\n      return TableMigrationUtil.listPartition(\n          partition.values,\n          partition.uri,\n          partition.format,\n          spec,\n          conf.get(),\n          metricsConfig,\n          mapping,\n          service);\n    } catch (RuntimeException e) {\n      if (ignoreMissingFiles && e.getCause() instanceof FileNotFoundException) {\n        LOG.warn(\"Ignoring FileNotFoundException when listing partition of {}\", partition, e);\n        return Collections.emptyList();\n      } else {\n        throw e;\n      }\n    }\n  }\n\n  private static SparkPartition toSparkPartition(\n      CatalogTablePartition partition, CatalogTable table) {\n    Option<URI> locationUri = partition.storage().locationUri();\n    Option<String> serde = partition.storage().serde();\n\n    Preconditions.checkArgument(locationUri.nonEmpty(), \"Partition URI should be defined\");\n    Preconditions.checkArgument(\n        serde.nonEmpty() || table.provider().nonEmpty(), \"Partition format should be defined\");\n\n    String uri = Util.uriToString(locationUri.get());\n    String format = serde.nonEmpty() ? serde.get() : table.provider().get();","sourceCodeStart":294,"sourceCodeEnd":330,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java#L294-L330","documentation":"When importing a Spark table partition, listing its files can fail with an underlying FileNotFoundException. If the import was configured with ignoreMissingFiles (spark.metadata.delete-after-commit.enabled-style option via SparkTableUtil.importSparkTable's check/ignore flag), the exception is swallowed, a warning is logged, and an empty file list is returned instead of failing the import.","triggerScenarios":"Calling SparkTableUtil.listPartition / filesToImport on a partition directory that was removed or not yet written while ignoreMissingFiles is enabled, causing the underlying Spark listing job to fail with FileNotFoundException as its cause.","commonSituations":"Migrating a live Hive/Spark table that is being concurrently compacted or rewritten; stale partition metadata pointing at deleted S3/HDFS paths; eventual-consistency lag on object stores.","solutions":["If missing partitions should fail the migration, disable the ignore-missing-files option so the FileNotFoundException propagates.","Verify the partition path exists and is readable in the underlying filesystem before importing (fs.exists / s3 ls).","Re-run the import after concurrent jobs (compaction, overwrite) finish to capture the missing files.","Check that the source table metadata (partition location) is fresh — refresh Hive metastore cache if stale."],"exampleFix":"// before: import fails because files vanished mid-migration\nSparkTableUtil.importSparkTable(spark, sourceTable, targetTable, stagingDir);\n// after: pass option to tolerate concurrently-deleted partitions\nMap<String, String> partitionFilter = Collections.emptyMap();\nSparkTableUtil.importSparkTable(spark, sourceTable, targetTable, stagingDir,\n    /* checkDeleteFiles */ true); // enables ignoring missing files","handlingStrategy":"validation","validationCode":"if (!fs.exists(new Path(partition.getLocation())) && !ignoreMissingFiles) {\n  throw new IllegalStateException(\"Partition path missing: \" + partition.getLocation());\n}","typeGuard":null,"tryCatchPattern":"try { files = SparkTableUtil.listPartition(...); } catch (RuntimeException e) {\n  if (!(e.getCause() instanceof FileNotFoundException)) throw e;\n  files = Collections.emptyList();\n}","preventionTips":["Quiesce concurrent compaction/overwrite jobs during table migration.","Pre-check partition paths in the metastore before import.","Decide explicitly whether missing partitions should be skipped or fatal and set the ignore flag accordingly."],"tags":["spark","migration","file-not-found","concurrency"],"backgroundTag":"file-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}