apache/beam · error · RuntimeException

Encountered an error when creating a RecordWriter for table

Error message

Encountered an error when creating a RecordWriter for table '{}', partition {}.

What it means

Beam's Iceberg RecordWriterManager wraps any IOException thrown while opening an Avro/Parquet writer for a table partition into a RuntimeException, preserving the table identifier and partition key as context. It means the underlying writer (FileAppender) could not be created, usually because the destination filesystem or catalog path is not writable or the write properties are invalid. The original IOException is attached as the cause and contains the true root reason.

Source

Thrown at sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/RecordWriterManager.java:206

    }

    private RecordWriter createWriter(PartitionKey partitionKey) {
      // keep track of how many writers we opened for each destination-partition path
      // use this as a prefix to differentiate the new path.
      // this avoids overwriting a data file written by a previous writer in this destination state.
      int recordIndex = writerCounts.merge(partitionKey, 1, Integer::sum);
      try {
        RecordWriter writer =
            new RecordWriter(
                table,
                icebergDestination.getFileFormat(),
                filePrefix + "_" + stateToken + "_" + recordIndex,
                partitionKey,
                writeProperties);
        openWriters++;
        return writer;
      } catch (IOException e) {
        throw new RuntimeException(
            String.format(
                "Encountered an error when creating a RecordWriter for table '%s', partition %s.",
                icebergDestination.getTableIdentifier(), partitionKey),
            e);
      }
    }
  }

  /**
   * Returns an equivalent partition path that is made up of partition data. Needed to reconstruct a
   * {@link DataFile}.
   */
  @VisibleForTesting
  static String getPartitionDataPath(
      String partitionPath, Map<String, PartitionField> partitionFieldMap) {
    if (partitionPath.isEmpty() || partitionFieldMap.isEmpty()) {
      return partitionPath;
    }

View on GitHub (pinned to 12126d8942)

Solutions

Inspect the wrapped IOException cause for the real reason (permission denied, path missing, bad config).
Verify the destination path/location is writable by the runner's credentials (test with a plain file write).
Check writeProperties passed to the destination for valid Iceberg write keys (write.format, write.parquet.compression-codec).
Ensure the table's location exists or can be auto-created by the catalog.
If transient (cloud storage), add retry/backoff around the pipeline stage or rerun the failing bundle.

Example fix

// before
// writeProperties.put("write.parquet.compression-codec", "zstd5"); // unsupported codec
// after
writeProperties.put("write.parquet.compression-codec", "zstd"); // valid codec

Defensive patterns

Strategy: try-catch

Validate before calling

// preflight: ensure destination location is writable
import org.apache.beam.sdk.io.fs.ResourceId;
// verify table location permissions before launching the pipeline
// e.g. attempt a small write/delete to table.location() with the runner's credentials

Try / catch

try {
  sink.write(records);
} catch (RuntimeException e) {
  if (e.getCause() instanceof IOException) {
    // inspect cause: permission, path, or writeProperties problem
    throw new PipelineStateException("RecordWriter creation failed for partition", e);
  }
  throw e;
}

Prevention

Validate writeProperties keys against Iceberg's documented write.* properties before submitting the job.
Pre-create and permission the table location for the runner's service account.
Include retry with backoff for transient cloud storage IOExceptions.
Log table identifier and partition key on failure for fast triage.

When it happens

Trigger: RecordWriterManager.createWriter() calls Iceberg's Avro/Parquet writer builder (FileAppenderFactory/writeProperties) and the underlying appender.open() throws IOException, e.g. unreadable or non-existent output directory, bad compression/codec property in writeProperties, or a filesystem outage.

Common situations: Writing to a partition path the service account cannot create; wrong or unsupported write.format/compression property passed via writeProperties; GCS/S3/HDFS transient failures or missing credentials; partition key derived from malformed data.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/289defb169b371da. Report an issue: GitHub.