{"record":{"id":"ccc5f92744f933c6","repo":"apache/iceberg","slug":"failed-to-create-parquet-file","errorCode":null,"errorMessage":"Failed to create Parquet file","messagePattern":"Failed to create Parquet file","errorType":"exception","errorClass":"UncheckedIOException","httpStatus":null,"severity":"critical","filePath":"parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java","lineNumber":128,"sourceCode":"    startRowGroup();\n  }\n\n  private void ensureWriterInitialized() {\n    if (writer == null) {\n      try {\n        this.writer =\n            new ParquetFileWriter(\n                ParquetIO.file(output, conf),\n                parquetSchema,\n                writeMode,\n                targetRowGroupSize,\n                0,\n                columnIndexTruncateLength,\n                ParquetProperties.DEFAULT_STATISTICS_TRUNCATE_LENGTH,\n                ParquetProperties.DEFAULT_PAGE_WRITE_CHECKSUM_ENABLED,\n                fileEncryptor);\n      } catch (IOException e) {\n        throw new UncheckedIOException(\"Failed to create Parquet file\", e);\n      }\n\n      try {\n        writer.start();\n      } catch (IOException e) {\n        throw new UncheckedIOException(\"Failed to start Parquet file writer\", e);\n      }\n    }\n  }\n\n  @Override\n  public void add(T value) {\n    recordCount += 1;\n    if (trackUncompressedSize) {\n      writeTracked(value);\n    } else {\n      model.write(0, value);\n    }","sourceCodeStart":110,"sourceCodeEnd":146,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java#L110-L146","documentation":"ParquetWriter.ensureWriterInitialized() creates the underlying parquet-mr ParquetFileWriter lazily (on first flushRowGroup). If the parquet-mr writer construction throws IOException, it is wrapped in UncheckedIOException with 'Failed to create Parquet file'. This separates file-creation failures from write failures and surfaces them as runtime exceptions.","triggerScenarios":"flushRowGroup -> ensureWriterInitialized when the OutputFile cannot be opened: missing output directory, no permission, unsupported/unresolvable filesystem scheme, or encryption setup errors (fileEncryptor) rejected by parquet-mr.","commonSituations":"Target path's parent directory doesn't exist (common on HDFS/S3 writes without directory creation); wrong filesystem scheme in the location URI; table location permissions revoked mid-task; misconfigured parquet encryption keys.","solutions":["Verify the output location is writable and its parent directory exists before starting the task; check the wrapped IOException cause","Fix the table/metadata location URI scheme so the correct filesystem is resolved","Review parquet encryption configuration (fileEncryptor) keys/KMS access if encryption is enabled"],"exampleFix":"// before: writing to a location whose parent does not exist\nOutputFile out = io.newOutputFile(\"s3://bucket/table/data/part-0000.parquet\");\n// after: create parent dirs first (or use a location that exists)\nio.newInputFile(\"s3://bucket/table\").location(); // ensure table dir exists; for HDFS:\nhdfs.mkdirs(new Path(parent));","handlingStrategy":"try-catch","validationCode":"// before starting the task:\nPreconditions.checkArgument(io.newInputFile(parentDir).exists(),\n    \"Output parent directory missing: %s\", parentDir);\n// plus verify the OutputFile can be created with a probe where cheap","typeGuard":null,"tryCatchPattern":"try {\n  writer.flushRowGroup();\n} catch (UncheckedIOException e) {\n  if (\"Failed to create Parquet file\".equals(e.getMessage())) {\n    LOG.error(\"Parquet file creation failed; check location/permissions/encryption\", e.getCause());\n  }\n  throw e;\n}","preventionTips":["Ensure output parent directories exist (or use a FileIO that creates them)","Verify location URI schemes resolve to a configured filesystem","Check parquet encryption keys/KMS availability when encryption is enabled","Smoke-test file creation in a preflight check before writing large data"],"tags":["parquet","io","file-creation","lazy-initialization"],"backgroundTag":"file-write-failed","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}