What causes "Failed to create Parquet file" in apache/iceberg?

Verify the output location is writable and its parent directory exists before starting the task; check the wrapped IOException cause

apache/iceberg · critical · UncheckedIOException

Failed to create Parquet file

Error message

Failed to create Parquet file

What it means

ParquetWriter.ensureWriterInitialized() creates the underlying parquet-mr ParquetFileWriter lazily (on first flushRowGroup). If the parquet-mr writer construction throws IOException, it is wrapped in UncheckedIOException with 'Failed to create Parquet file'. This separates file-creation failures from write failures and surfaces them as runtime exceptions.

Solutions

Verify the output location is writable and its parent directory exists before starting the task; check the wrapped IOException cause
Fix the table/metadata location URI scheme so the correct filesystem is resolved
Review parquet encryption configuration (fileEncryptor) keys/KMS access if encryption is enabled

Example fix

// before: writing to a location whose parent does not exist
OutputFile out = io.newOutputFile("s3://bucket/table/data/part-0000.parquet");
// after: create parent dirs first (or use a location that exists)
io.newInputFile("s3://bucket/table").location(); // ensure table dir exists; for HDFS:
hdfs.mkdirs(new Path(parent));

Defensive patterns

Strategy: try-catch

Validate before calling

// before starting the task:
Preconditions.checkArgument(io.newInputFile(parentDir).exists(),
    "Output parent directory missing: %s", parentDir);
// plus verify the OutputFile can be created with a probe where cheap

Try / catch

try {
  writer.flushRowGroup();
} catch (UncheckedIOException e) {
  if ("Failed to create Parquet file".equals(e.getMessage())) {
    LOG.error("Parquet file creation failed; check location/permissions/encryption", e.getCause());
  }
  throw e;
}

Prevention

Ensure output parent directories exist (or use a FileIO that creates them)
Verify location URI schemes resolve to a configured filesystem
Check parquet encryption keys/KMS availability when encryption is enabled
Smoke-test file creation in a preflight check before writing large data

When it happens

Trigger: flushRowGroup -> ensureWriterInitialized when the OutputFile cannot be opened: missing output directory, no permission, unsupported/unresolvable filesystem scheme, or encryption setup errors (fileEncryptor) rejected by parquet-mr.

Common situations: Target path's parent directory doesn't exist (common on HDFS/S3 writes without directory creation); wrong filesystem scheme in the location URI; table location permissions revoked mid-task; misconfigured parquet encryption keys.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ccc5f92744f933c6. Report an issue: GitHub.

Appendix: source

Thrown at parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java:128

    startRowGroup();
  }

  private void ensureWriterInitialized() {
    if (writer == null) {
      try {
        this.writer =
            new ParquetFileWriter(
                ParquetIO.file(output, conf),
                parquetSchema,
                writeMode,
                targetRowGroupSize,
                0,
                columnIndexTruncateLength,
                ParquetProperties.DEFAULT_STATISTICS_TRUNCATE_LENGTH,
                ParquetProperties.DEFAULT_PAGE_WRITE_CHECKSUM_ENABLED,
                fileEncryptor);
      } catch (IOException e) {
        throw new UncheckedIOException("Failed to create Parquet file", e);
      }

      try {
        writer.start();
      } catch (IOException e) {
        throw new UncheckedIOException("Failed to start Parquet file writer", e);
      }
    }
  }

  @Override
  public void add(T value) {
    recordCount += 1;
    if (trackUncompressedSize) {
      writeTracked(value);
    } else {
      model.write(0, value);
    }

View on GitHub (pinned to 86d9c8fc54)