{"record":{"id":"289defb169b371da","repo":"apache/beam","slug":"encountered-an-error-when-creating-a-recordwriter-for-table","errorCode":null,"errorMessage":"Encountered an error when creating a RecordWriter for table '{}', partition {}.","messagePattern":"Encountered an error when creating a RecordWriter for table '(.+?)', partition (.+?)\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriterManager.java","lineNumber":206,"sourceCode":"    }\n\n    private RecordWriter createWriter(PartitionKey partitionKey) {\n      // keep track of how many writers we opened for each destination-partition path\n      // use this as a prefix to differentiate the new path.\n      // this avoids overwriting a data file written by a previous writer in this destination state.\n      int recordIndex = writerCounts.merge(partitionKey, 1, Integer::sum);\n      try {\n        RecordWriter writer =\n            new RecordWriter(\n                table,\n                icebergDestination.getFileFormat(),\n                filePrefix + \"_\" + stateToken + \"_\" + recordIndex,\n                partitionKey,\n                writeProperties);\n        openWriters++;\n        return writer;\n      } catch (IOException e) {\n        throw new RuntimeException(\n            String.format(\n                \"Encountered an error when creating a RecordWriter for table '%s', partition %s.\",\n                icebergDestination.getTableIdentifier(), partitionKey),\n            e);\n      }\n    }\n  }\n\n  /**\n   * Returns an equivalent partition path that is made up of partition data. Needed to reconstruct a\n   * {@link DataFile}.\n   */\n  @VisibleForTesting\n  static String getPartitionDataPath(\n      String partitionPath, Map<String, PartitionField> partitionFieldMap) {\n    if (partitionPath.isEmpty() || partitionFieldMap.isEmpty()) {\n      return partitionPath;\n    }","sourceCodeStart":188,"sourceCodeEnd":224,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriterManager.java#L188-L224","documentation":"Beam's Iceberg RecordWriterManager wraps any IOException thrown while opening an Avro/Parquet writer for a table partition into a RuntimeException, preserving the table identifier and partition key as context. It means the underlying writer (FileAppender) could not be created, usually because the destination filesystem or catalog path is not writable or the write properties are invalid. The original IOException is attached as the cause and contains the true root reason.","triggerScenarios":"RecordWriterManager.createWriter() calls Iceberg's Avro/Parquet writer builder (FileAppenderFactory/writeProperties) and the underlying appender.open() throws IOException, e.g. unreadable or non-existent output directory, bad compression/codec property in writeProperties, or a filesystem outage.","commonSituations":"Writing to a partition path the service account cannot create; wrong or unsupported write.format/compression property passed via writeProperties; GCS/S3/HDFS transient failures or missing credentials; partition key derived from malformed data.","solutions":["Inspect the wrapped IOException cause for the real reason (permission denied, path missing, bad config).","Verify the destination path/location is writable by the runner's credentials (test with a plain file write).","Check writeProperties passed to the destination for valid Iceberg write keys (write.format, write.parquet.compression-codec).","Ensure the table's location exists or can be auto-created by the catalog.","If transient (cloud storage), add retry/backoff around the pipeline stage or rerun the failing bundle."],"exampleFix":"// before\n// writeProperties.put(\"write.parquet.compression-codec\", \"zstd5\"); // unsupported codec\n// after\nwriteProperties.put(\"write.parquet.compression-codec\", \"zstd\"); // valid codec","handlingStrategy":"try-catch","validationCode":"// preflight: ensure destination location is writable\nimport org.apache.beam.sdk.io.fs.ResourceId;\n// verify table location permissions before launching the pipeline\n// e.g. attempt a small write/delete to table.location() with the runner's credentials","typeGuard":null,"tryCatchPattern":"try {\n  sink.write(records);\n} catch (RuntimeException e) {\n  if (e.getCause() instanceof IOException) {\n    // inspect cause: permission, path, or writeProperties problem\n    throw new PipelineStateException(\"RecordWriter creation failed for partition\", e);\n  }\n  throw e;\n}","preventionTips":["Validate writeProperties keys against Iceberg's documented write.* properties before submitting the job.","Pre-create and permission the table location for the runner's service account.","Include retry with backoff for transient cloud storage IOExceptions.","Log table identifier and partition key on failure for fast triage."],"tags":["iceberg","java","file-io","writer","apache-beam"],"backgroundTag":"file-write-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}