{"record":{"id":"3ad654db49254641","repo":"apache/iceberg","slug":"ignoring-filenotfoundexception-when-listing-partit-3ad654","errorCode":null,"errorMessage":"Ignoring FileNotFoundException when listing partition of {}","messagePattern":"Ignoring FileNotFoundException when listing partition of (.+?)","errorType":"console","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java","lineNumber":185,"sourceCode":"      PartitionSpec spec,\n      SerializableConfiguration conf,\n      MetricsConfig metricsConfig,\n      NameMapping mapping,\n      boolean ignoreMissingFiles,\n      ExecutorService service) {\n    try {\n      return TableMigrationUtil.listPartition(\n          partition.values,\n          partition.uri,\n          partition.format,\n          spec,\n          conf.get(),\n          metricsConfig,\n          mapping,\n          service);\n    } catch (RuntimeException e) {\n      if (ignoreMissingFiles && e.getCause() instanceof FileNotFoundException) {\n        LOG.warn(\"Ignoring FileNotFoundException when listing partition of {}\", partition, e);\n        return Collections.emptyList();\n      } else {\n        throw e;\n      }\n    }\n  }\n\n  private static SparkPartition toSparkPartition(\n      CatalogTablePartition partition, CatalogTable table) {\n    Option<URI> locationUri = partition.storage().locationUri();\n    Option<String> partitionSerde = partition.storage().serde();\n\n    Preconditions.checkArgument(locationUri.nonEmpty(), \"Partition URI should be defined\");\n    Preconditions.checkArgument(\n        partitionSerde.nonEmpty() || table.provider().nonEmpty(),\n        \"Partition format should be defined\");\n\n    String uri = Util.uriToString(locationUri.get());","sourceCodeStart":167,"sourceCodeEnd":203,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java#L167-L203","documentation":"SparkTableUtil.listPartition lists data files of a Hive-style partition. When ignoreMissingFiles is enabled and the underlying listing fails with a FileNotFoundException (as the cause of the RuntimeException), the warning is logged and an empty list is returned instead of failing — matching Spark's spark.sql.files.ignoreMissingFiles semantics. Otherwise the exception is rethrown.","triggerScenarios":"Importing/listing an Iceberg partition from existing Hive data (SparkTableUtil.listPartition / importSparkTable) where partition files were deleted or not yet fully written on the storage system while ignoreMissingFiles is true.","commonSituations":"Concurrent jobs deleting files during import; race between partition rewrite and listing; listing a stale partition path in a fast-moving table; S3 eventual consistency artifacts (legacy).","solutions":["Decide whether missing files are expected: if not, disable ignoreMissingFiles (spark.sql.files.ignoreMissingFiles=false) to fail loudly.","If expected, keep the current behavior — the empty partition list is intentional and safe for import.","Re-run the import/listing after concurrent writers finish to capture the missing files.","Verify the partition path and table location are correct before re-running."],"exampleFix":"// before: silently skips missing files\nspark.conf.set(\"spark.sql.files.ignoreMissingFiles\", \"true\")\n\n// after: fail fast when files disappear\nspark.conf.set(\"spark.sql.files.ignoreMissingFiles\", \"false\")","handlingStrategy":"validation","validationCode":"// Confirm all expected partition files exist before import\nspark.conf().get(\"spark.sql.files.ignoreMissingFiles\", \"false\") // keep false to fail on missing files\n\nboolean exists = table.io().newInputFile(partitionDir).exists();\nPreconditions.checkArgument(exists, \"Partition path missing: \" + partitionDir);","typeGuard":null,"tryCatchPattern":"try {\n  List<SparkDataFile> files = SparkTableUtil.listPartition(spark, table, partition, spec);\n} catch (RuntimeException e) {\n  if (!(e.getCause() instanceof FileNotFoundException)) throw e;\n  // decide: retry or treat as empty\n}","preventionTips":["Avoid importing while concurrent jobs delete/rewrite the source partition","Keep spark.sql.files.ignoreMissingFiles=false when missing files are unexpected","Verify partition paths before listing","Re-run imports after concurrent writers complete"],"tags":["spark","file-not-found","partition","listing"],"backgroundTag":"file-not-found","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}