apache/iceberg · error · UnsupportedOperationException

Cannot serialize a Hadoop input file: " + location()

Error message

Cannot serialize a Hadoop input file: " + location()

What it means

HadoopInputFile.serializeConfWith throws UnsupportedOperationException because Hadoop-backed InputFiles cannot capture a serializable Configuration supplier. SerializableTable serializes its FileIO instead, and Hadoop file objects are not designed to carry their Hadoop conf across serialization boundaries.

Source

Thrown at core/src/main/java/org/apache/iceberg/hadoop/HadoopInputFile.java:201

  public SeekableInputStream newStream() {
    try {
      return HadoopStreams.wrap(fs.open(path));
    } catch (FileNotFoundException e) {
      throw new NotFoundException(e, "Failed to open input stream for file: %s", path);
    } catch (IOException e) {
      throw new RuntimeIOException(e, "Failed to open input stream for file: %s", path);
    }
  }

  @Override
  public Configuration getConf() {
    return conf;
  }

  @Override
  public void serializeConfWith(
      Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {
    throw new UnsupportedOperationException("Cannot serialize a Hadoop input file: " + location());
  }

  public FileSystem getFileSystem() {
    return fs;
  }

  public FileStatus getStat() {
    return lazyStat();
  }

  public Path getPath() {
    return path;
  }

  public String[] getBlockLocations(long start, long end) {
    List<String> hosts = Lists.newArrayList();
    try {
      for (BlockLocation bl : fs.getFileBlockLocations(path, start, end)) {

View on GitHub (pinned to 86d9c8fc54)

Solutions

1. Do not call serializeConfWith on HadoopInputFile; rely on Hadoop's own Configuration propagation to executors
2. Use a non-Hadoop FileIO (HadoopFileIO configuration is passed via Hadoop conf; S3FileIO serializes its conf supplier) if you need serializable conf capture
3. Construct the HadoopInputFile on the executor from a serialized HadoopConf/SerializableConfiguration instead of serializing the file

Example fix

// before
hadoopInputFile.serializeConfWith(SerializableConfiguration::new); // throws
// after
// pass configuration via SerializableConfiguration when shipping to executors
SerializableConfiguration serConf = new SerializableConfiguration(hadoopConf);
// ... on executor: new HadoopInputFile(serConf.get(), path, stats);

if (inputFile instanceof HadoopInputFile) {
  throw new IllegalStateException("HadoopInputFile cannot be conf-serialized; ship Hadoop conf instead");
}

boolean supportsConfSerialization = !(file instanceof HadoopInputFile);

try { file.serializeConfWith(supplier); }
catch (UnsupportedOperationException e) { /* fall back to Hadoop conf propagation */ }

When it happens

Trigger: Explicitly calling serializeConfWith(...) on a HadoopInputFile, or any code path (e.g. SerializableTable/Flink/Spark serialization helpers) that invokes it on a Hadoop-backed input file.

Common situations: Distributed execution (Spark/Flink executors) where a task closure attempts to serialize the file object; custom code treating HadoopInputFile like InMemoryInputFile or S3InputFile which support conf serialization.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• Unsupported primitive type:
• Cannot set namespace properties " + namespace + " : setPrope
• Cannot remove properties " + namespace + " : removePropertie
• setConf is not implemented
• getConf is not implemented

AI-assisted analysis of apache/iceberg@86d9c8fc54 (2026-09-12). Data as JSON: /api/errors/0471172ec7f821a4. Report an issue: GitHub.