apache/iceberg · error · UncheckedIOException

Failed to create shredded variant writer

Error message

Failed to create shredded variant writer

What it means

When the schema contains variant columns and shredding is enabled, buildShreddedAppender() constructs a writer whose build() may throw IOException. That exception is wrapped in an UncheckedIOException with this message, indicating the shredded-variant Parquet writer could not be created (e.g. failure opening/initializing the output stream).

Solutions

  1. Check the wrapped IOException cause for the underlying file/IO error
  2. Verify the output file location is writable and the filesystem is healthy
  3. Ensure the output stream/FileIO is properly opened and not already closed
  4. Retry the write after fixing storage issues
Defensive patterns

Strategy: try-catch

Validate before calling

// pre-check output target writability before building the writer
Verify that the output file location exists and is writable before calling build()

Try / catch

try {
  return buildShreddedAppender();
} catch (UncheckedIOException e) {
  logger.error("Shredded variant writer creation failed", e.getCause());
  throw e;
}

Prevention

When it happens

Trigger: Writing a Parquet file with variant columns and shredVariants enabled when internal.build() throws IOException — typically due to the underlying output file/stream failing to open or initialize.

Common situations: Unwritable destination path, full disk, closed or failed output stream, Hadoop FileSystem errors during file creation while writing shredded variant data.

Understand the failure class

Background: "failed to write file", "Could not save figure", "Error saving remote file" — file write failed: causes and fixes across languages and libraries — this error's family across 38 libraries.

Related errors

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/ce9808a29ab453c9. Report an issue: GitHub.

Appendix: source

Thrown at parquet/src/main/java/org/apache/iceberg/parquet/ParquetFormatModel.java:318

      return new BufferedFileAppender<>(
          bufferSize,
          bufferedRows -> {
            Map<Integer, Type> shreddedTypes =
                variantAnalyzer.analyzeVariantColumns(bufferedRows, schema, engineSchema);
            LOG.debug(
                "Variant inference: rows={}, shredded fields={}",
                bufferedRows.size(),
                shreddedTypes.size());

            if (!shreddedTypes.isEmpty()) {
              internal.variantShreddingFunc((fieldId, name) -> shreddedTypes.get(fieldId));
            }

            try {
              return internal.build();
            } catch (IOException e) {
              throw new UncheckedIOException("Failed to create shredded variant writer", e);
            }
          },
          copyFunc);
    }

    private static boolean hasVariantColumns(Schema schema) {
      return schema != null
          && schema.columns().stream().anyMatch(field -> field.type().isVariantType());
    }
  }

  private static class ReadBuilderWrapper<R, D, S> implements ReadBuilder<D, S> {
    private final Parquet.ReadBuilder internal;
    private final ReaderFunction<R, S, MessageType> readerFunction;
    private final boolean isBatchReader;
    private S engineSchema;
    private Map<Integer, ?> idToConstant = ImmutableMap.of();

View on GitHub (pinned to 86d9c8fc54)