What causes "Failed to get statistics from writer" in apache/iceberg?

Inspect the underlying IOException cause; treat it as a symptom of the failed write — retry the whole write task so a healthy writer produces metrics.

apache/iceberg · error · RuntimeIOException

Failed to get statistics from writer

Error message

Failed to get statistics from writer

What it means

OrcMetrics.fromWriter computes metrics from an in-progress ORC Writer by asking it for its schema and accumulated statistics. Any IOException from writer.getStatistics() is wrapped in RuntimeIOException without a file path (the message does not include one because metrics may be gathered before/without a file location). This reflects a failure reading live writer state, not the data file.

Source

Thrown at orc/src/main/java/org/apache/iceberg/orc/OrcMetrics.java:119

          metricsConfig,
          mapping);
    } catch (IOException ioe) {
      throw new RuntimeIOException(ioe, "Failed to open file: %s", file.location());
    }
  }

  static Metrics fromWriter(
      Writer writer, Stream<FieldMetrics<?>> fieldMetricsStream, MetricsConfig metricsConfig) {
    try {
      return buildOrcMetrics(
          writer.getNumberOfRows(),
          writer.getSchema(),
          writer.getStatistics(),
          fieldMetricsStream,
          metricsConfig,
          null);
    } catch (IOException ioe) {
      throw new RuntimeIOException(ioe, "Failed to get statistics from writer");
    }
  }

  private static Metrics buildOrcMetrics(
      final long numOfRows,
      final TypeDescription orcSchema,
      final ColumnStatistics[] colStats,
      final Stream<FieldMetrics<?>> fieldMetricsStream,
      final MetricsConfig metricsConfig,
      final NameMapping mapping) {
    TypeDescription orcSchemaWithIds;
    if (ORCSchemaUtil.hasIds(orcSchema)) {
      orcSchemaWithIds = orcSchema;
    } else if (mapping != null) {
      orcSchemaWithIds = ORCSchemaUtil.applyNameMapping(orcSchema, mapping);
    } else {
      return new Metrics(numOfRows);
    }

View on GitHub (pinned to 86d9c8fc54)

Solutions

Inspect the underlying IOException cause; treat it as a symptom of the failed write — retry the whole write task so a healthy writer produces metrics.
Ensure fromWriter is only called while the Writer is open and healthy (typically inside the appender's close/commit flow).
Check storage backend health/permissions; if the writer already failed on an earlier I/O error, fix that root cause first.

Example fix

// before (writer in failed state)
Metrics metrics = OrcMetrics.fromWriter(writer, metricsStream, config); // RuntimeIOException

// after: only extract metrics from a successfully-written writer
try (FileAppender<Record> app = ORC.write(out).schema(schema).metricsConfig(config).build()) {
  app.addAll(records);
  // metrics are computed internally on close from the healthy writer
}

Defensive patterns

Strategy: retry

Try / catch

try {
  Metrics m = OrcMetrics.fromWriter(writer, metricsStream, config);
} catch (RuntimeIOException e) {
  LOG.error("Writer statistics unavailable: {}", e.getCause());
  throw e; // retry the entire write task
}

Prevention

Only extract metrics from a writer that has not failed on a prior I/O operation
Treat statistics failures as write-task failures and retry the whole task
Check storage backend health when finalizing ORC files

When it happens

Trigger: Calling OrcMetrics.fromWriter(writer, fieldMetricsStream, metricsConfig) during appender close/commit when the ORC Writer's internal statistics retrieval throws IOException (writer already failed/closed, backend error while reading buffered stripe stats).

Common situations: Commit path computing metrics after an earlier write failure left the writer in a bad state; storage-backend errors (S3/HDFS) while the ORC Writer materializes statistics during finalization.

Understand the failure class

Background: "failed to read file", EACCES, ENOENT and "could not read <path>" errors: when a program can't read a file from disk — this error's family across 49 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/3f2567e40b1b79b2. Report an issue: GitHub.