apache/iceberg · warning

Ignoring FileNotFoundException when listing partition of {}

Error message

Ignoring FileNotFoundException when listing partition of {}

What it means

When importing a Spark table partition, listing its files can fail with an underlying FileNotFoundException. If the import was configured with ignoreMissingFiles (spark.metadata.delete-after-commit.enabled-style option via SparkTableUtil.importSparkTable's check/ignore flag), the exception is swallowed, a warning is logged, and an empty file list is returned instead of failing the import.

Source

Thrown at spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:312

      PartitionSpec spec,
      SerializableConfiguration conf,
      MetricsConfig metricsConfig,
      NameMapping mapping,
      boolean ignoreMissingFiles,
      ExecutorService service) {
    try {
      return TableMigrationUtil.listPartition(
          partition.values,
          partition.uri,
          partition.format,
          spec,
          conf.get(),
          metricsConfig,
          mapping,
          service);
    } catch (RuntimeException e) {
      if (ignoreMissingFiles && e.getCause() instanceof FileNotFoundException) {
        LOG.warn("Ignoring FileNotFoundException when listing partition of {}", partition, e);
        return Collections.emptyList();
      } else {
        throw e;
      }
    }
  }

  private static SparkPartition toSparkPartition(
      CatalogTablePartition partition, CatalogTable table) {
    Option<URI> locationUri = partition.storage().locationUri();
    Option<String> serde = partition.storage().serde();

    Preconditions.checkArgument(locationUri.nonEmpty(), "Partition URI should be defined");
    Preconditions.checkArgument(
        serde.nonEmpty() || table.provider().nonEmpty(), "Partition format should be defined");

    String uri = Util.uriToString(locationUri.get());
    String format = serde.nonEmpty() ? serde.get() : table.provider().get();

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. If missing partitions should fail the migration, disable the ignore-missing-files option so the FileNotFoundException propagates.
2. Verify the partition path exists and is readable in the underlying filesystem before importing (fs.exists / s3 ls).
3. Re-run the import after concurrent jobs (compaction, overwrite) finish to capture the missing files.
4. Check that the source table metadata (partition location) is fresh — refresh Hive metastore cache if stale.

Example fix

// before: import fails because files vanished mid-migration
SparkTableUtil.importSparkTable(spark, sourceTable, targetTable, stagingDir);
// after: pass option to tolerate concurrently-deleted partitions
Map<String, String> partitionFilter = Collections.emptyMap();
SparkTableUtil.importSparkTable(spark, sourceTable, targetTable, stagingDir,
    /* checkDeleteFiles */ true); // enables ignoring missing files

if (!fs.exists(new Path(partition.getLocation())) && !ignoreMissingFiles) {
  throw new IllegalStateException("Partition path missing: " + partition.getLocation());
}

try { files = SparkTableUtil.listPartition(...); } catch (RuntimeException e) {
  if (!(e.getCause() instanceof FileNotFoundException)) throw e;
  files = Collections.emptyList();
}

When it happens

Trigger: Calling SparkTableUtil.listPartition / filesToImport on a partition directory that was removed or not yet written while ignoreMissingFiles is enabled, causing the underlying Spark listing job to fail with FileNotFoundException as its cause.

Common situations: Migrating a live Hive/Spark table that is being concurrently compacted or rewritten; stale partition metadata pointing at deleted S3/HDFS paths; eventual-consistency lag on object stores.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co
• Cannot find a partition spec in Iceberg table %s that matche
• Cannot commit rewrite because of a ValidationException or Co
• Cannot commit rewrite because of a ValidationException or Co

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/03bec785285f3d84. Report an issue: GitHub.