apache/iceberg · warning

Ignoring FileNotFoundException when listing partition of {}

Error message

Ignoring FileNotFoundException when listing partition of {}

What it means

When listing a Hive partition's files during SparkTableUtil.listPartition, a FileNotFoundException wrapped in a RuntimeException can occur if the partition directory/files disappeared between metadata listing and file listing. If ignoreMissingFiles is set, the warning is logged and an empty list is returned instead of failing the import/migration.

Source

Thrown at spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:184

      PartitionSpec spec,
      SerializableConfiguration conf,
      MetricsConfig metricsConfig,
      NameMapping mapping,
      boolean ignoreMissingFiles,
      ExecutorService service) {
    try {
      return TableMigrationUtil.listPartition(
          partition.values,
          partition.uri,
          partition.format,
          spec,
          conf.get(),
          metricsConfig,
          mapping,
          service);
    } catch (RuntimeException e) {
      if (ignoreMissingFiles && e.getCause() instanceof FileNotFoundException) {
        LOG.warn("Ignoring FileNotFoundException when listing partition of {}", partition, e);
        return Collections.emptyList();
      } else {
        throw e;
      }
    }
  }

  private static SparkPartition toSparkPartition(
      CatalogTablePartition partition, CatalogTable table) {
    Option<URI> locationUri = partition.storage().locationUri();
    Option<String> partitionSerde = partition.storage().serde();

    Preconditions.checkArgument(locationUri.nonEmpty(), "Partition URI should be defined");
    Preconditions.checkArgument(
        partitionSerde.nonEmpty() || table.provider().nonEmpty(),
        "Partition format should be defined");

    String uri = Util.uriToString(locationUri.get());

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Set ignore-missing-files to true if missing partition data is acceptable, then re-check counts after migration
2. Restore/recreate the missing partition data in the source table before re-running the import
3. Verify partition locations with a filesystem listing to find which partitions are actually gone

Example fix

// before
spark.sql.iceberg.ignore-missing-files=false
// after
spark.sql.iceberg.ignore-missing-files=true

// before migration, confirm partition locations exist
for (String loc : partitionLocations) {
  if (!fs.exists(new Path(loc))) missing.add(loc);
}

When it happens

Trigger: Migrating or importing a Hive table where a partition's underlying files were deleted (or not yet written) after the metastore recorded the partition, with spark.sql.iceberg.ignore-missing-files / ignoreMissingFiles=true.

Common situations: Racing jobs that drop or overwrite Hive partitions during a migration; partitions whose data was manually removed from storage; HDFS/S3 eventual consistency issues.

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Cannot find a partition spec in Iceberg table %s that matche
• Unexpected data type in partition filters: ${dataType}
• Cannot find a partition spec in Iceberg table %s that matche
• Cannot write using unsupported transforms: %s
• Cannot find a partition spec in Iceberg table %s that matche

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/cf1fd72447872702. Report an issue: GitHub.