apache/iceberg · warning

Ignoring FileNotFoundException when listing partition of {}

Error message

Ignoring FileNotFoundException when listing partition of {}

What it means

SparkTableUtil.listPartition lists data files of a Hive-style partition. When ignoreMissingFiles is enabled and the underlying listing fails with a FileNotFoundException (as the cause of the RuntimeException), the warning is logged and an empty list is returned instead of failing — matching Spark's spark.sql.files.ignoreMissingFiles semantics. Otherwise the exception is rethrown.

Source

Thrown at spark/v4.0/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:185

      PartitionSpec spec,
      SerializableConfiguration conf,
      MetricsConfig metricsConfig,
      NameMapping mapping,
      boolean ignoreMissingFiles,
      ExecutorService service) {
    try {
      return TableMigrationUtil.listPartition(
          partition.values,
          partition.uri,
          partition.format,
          spec,
          conf.get(),
          metricsConfig,
          mapping,
          service);
    } catch (RuntimeException e) {
      if (ignoreMissingFiles && e.getCause() instanceof FileNotFoundException) {
        LOG.warn("Ignoring FileNotFoundException when listing partition of {}", partition, e);
        return Collections.emptyList();
      } else {
        throw e;
      }
    }
  }

  private static SparkPartition toSparkPartition(
      CatalogTablePartition partition, CatalogTable table) {
    Option<URI> locationUri = partition.storage().locationUri();
    Option<String> partitionSerde = partition.storage().serde();

    Preconditions.checkArgument(locationUri.nonEmpty(), "Partition URI should be defined");
    Preconditions.checkArgument(
        partitionSerde.nonEmpty() || table.provider().nonEmpty(),
        "Partition format should be defined");

    String uri = Util.uriToString(locationUri.get());

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Decide whether missing files are expected: if not, disable ignoreMissingFiles (spark.sql.files.ignoreMissingFiles=false) to fail loudly.
2. If expected, keep the current behavior — the empty partition list is intentional and safe for import.
3. Re-run the import/listing after concurrent writers finish to capture the missing files.
4. Verify the partition path and table location are correct before re-running.

Example fix

// before: silently skips missing files
spark.conf.set("spark.sql.files.ignoreMissingFiles", "true")

// after: fail fast when files disappear
spark.conf.set("spark.sql.files.ignoreMissingFiles", "false")

// Confirm all expected partition files exist before import
spark.conf().get("spark.sql.files.ignoreMissingFiles", "false") // keep false to fail on missing files

boolean exists = table.io().newInputFile(partitionDir).exists();
Preconditions.checkArgument(exists, "Partition path missing: " + partitionDir);

try {
  List<SparkDataFile> files = SparkTableUtil.listPartition(spark, table, partition, spec);
} catch (RuntimeException e) {
  if (!(e.getCause() instanceof FileNotFoundException)) throw e;
  // decide: retry or treat as empty
}

When it happens

Trigger: Importing/listing an Iceberg partition from existing Hive data (SparkTableUtil.listPartition / importSparkTable) where partition files were deleted or not yet fully written on the storage system while ignoreMissingFiles is true.

Common situations: Concurrent jobs deleting files during import; race between partition rewrite and listing; listing a stale partition path in a fast-moving table; S3 eventual consistency artifacts (legacy).

Understand the failure class

Background: "File not found" and ENOENT errors: why libraries can't find a file that should exist — this error's family across 50 libraries.

Related errors

• Could not list sub directories, reached maximum depth:
• Ignoring FileNotFoundException when listing partition of {}
• Ignoring FileNotFoundException when listing partition of {}
• Failed to create file: %s
• Ignoring missing files is not supported

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3ad654db49254641. Report an issue: GitHub.