{"record":{"id":"0471172ec7f821a4","repo":"apache/iceberg","slug":"cannot-serialize-a-hadoop-input-file-location","errorCode":null,"errorMessage":"Cannot serialize a Hadoop input file: \" + location()","messagePattern":"Cannot serialize a Hadoop input file: \" \\+ location\\(\\)","errorType":"exception","errorClass":"UnsupportedOperationException","httpStatus":null,"severity":"error","filePath":"core/src/main/java/org/apache/iceberg/hadoop/HadoopInputFile.java","lineNumber":201,"sourceCode":"  public SeekableInputStream newStream() {\n    try {\n      return HadoopStreams.wrap(fs.open(path));\n    } catch (FileNotFoundException e) {\n      throw new NotFoundException(e, \"Failed to open input stream for file: %s\", path);\n    } catch (IOException e) {\n      throw new RuntimeIOException(e, \"Failed to open input stream for file: %s\", path);\n    }\n  }\n\n  @Override\n  public Configuration getConf() {\n    return conf;\n  }\n\n  @Override\n  public void serializeConfWith(\n      Function<Configuration, SerializableSupplier<Configuration>> confSerializer) {\n    throw new UnsupportedOperationException(\"Cannot serialize a Hadoop input file: \" + location());\n  }\n\n  public FileSystem getFileSystem() {\n    return fs;\n  }\n\n  public FileStatus getStat() {\n    return lazyStat();\n  }\n\n  public Path getPath() {\n    return path;\n  }\n\n  public String[] getBlockLocations(long start, long end) {\n    List<String> hosts = Lists.newArrayList();\n    try {\n      for (BlockLocation bl : fs.getFileBlockLocations(path, start, end)) {","sourceCodeStart":183,"sourceCodeEnd":219,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/core/src/main/java/org/apache/iceberg/hadoop/HadoopInputFile.java#L183-L219","documentation":"HadoopInputFile.serializeConfWith throws UnsupportedOperationException because Hadoop-backed InputFiles cannot capture a serializable Configuration supplier. SerializableTable serializes its FileIO instead, and Hadoop file objects are not designed to carry their Hadoop conf across serialization boundaries.","triggerScenarios":"Explicitly calling serializeConfWith(...) on a HadoopInputFile, or any code path (e.g. SerializableTable/Flink/Spark serialization helpers) that invokes it on a Hadoop-backed input file.","commonSituations":"Distributed execution (Spark/Flink executors) where a task closure attempts to serialize the file object; custom code treating HadoopInputFile like InMemoryInputFile or S3InputFile which support conf serialization.","solutions":["Do not call serializeConfWith on HadoopInputFile; rely on Hadoop's own Configuration propagation to executors","Use a non-Hadoop FileIO (HadoopFileIO configuration is passed via Hadoop conf; S3FileIO serializes its conf supplier) if you need serializable conf capture","Construct the HadoopInputFile on the executor from a serialized HadoopConf/SerializableConfiguration instead of serializing the file"],"exampleFix":"// before\nhadoopInputFile.serializeConfWith(SerializableConfiguration::new); // throws\n// after\n// pass configuration via SerializableConfiguration when shipping to executors\nSerializableConfiguration serConf = new SerializableConfiguration(hadoopConf);\n// ... on executor: new HadoopInputFile(serConf.get(), path, stats);","handlingStrategy":"type-guard","validationCode":"if (inputFile instanceof HadoopInputFile) {\n  throw new IllegalStateException(\"HadoopInputFile cannot be conf-serialized; ship Hadoop conf instead\");\n}","typeGuard":"boolean supportsConfSerialization = !(file instanceof HadoopInputFile);","tryCatchPattern":"try { file.serializeConfWith(supplier); }\ncatch (UnsupportedOperationException e) { /* fall back to Hadoop conf propagation */ }","preventionTips":["Never serialize Hadoop-backed file objects into task closures","Distribute configuration via SerializableConfiguration or the FileIO's own mechanism","Only call serializeConfWith on FileIO implementations that document support"],"tags":["hadoop","serialization","unsupported"],"backgroundTag":"unsupported-operation","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}