{"record":{"id":"670f2dfe9ac84a20","repo":"apache/iceberg","slug":"configuration-was-not-serialized-on-purpose-but-wa","errorCode":null,"errorMessage":"Configuration was not serialized on purpose but was not set manually either","messagePattern":"Configuration was not serialized on purpose but was not set manually either","errorType":"exception","errorClass":"IllegalStateException","httpStatus":null,"severity":"error","filePath":"mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java","lineNumber":218,"sourceCode":"    }\n  }\n\n  @Override\n  public RecordReader<Void, T> createRecordReader(InputSplit split, TaskAttemptContext context) {\n    return new IcebergRecordReader<>();\n  }\n\n  private static class NonSerializingConfig implements Serializable {\n\n    private final transient Configuration conf;\n\n    NonSerializingConfig(Configuration conf) {\n      this.conf = conf;\n    }\n\n    public Configuration get() {\n      if (conf == null) {\n        throw new IllegalStateException(\n            \"Configuration was not serialized on purpose but was not set manually either\");\n      }\n\n      return conf;\n    }\n  }\n\n  private static final class IcebergRecordReader<T> extends RecordReader<Void, T> {\n\n    private TaskAttemptContext context;\n    private Schema tableSchema;\n    private Schema expectedSchema;\n    private String nameMapping;\n    private boolean reuseContainers;\n    private boolean caseSensitive;\n    private Iterator<FileScanTask> tasks;\n    private T current;\n    private CloseableIterator<T> currentIterator;","sourceCodeStart":200,"sourceCodeEnd":236,"githubUrl":"https://github.com/apache/iceberg/blob/86d9c8fc543e7c56c9f624eb725f76c9baff9570/mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java#L200-L236","documentation":"Iceberg's mapreduce IcebergInputFormat wraps the Hadoop Configuration in a NonSerializingConfig that deliberately refuses to serialize the Configuration object (Configurations are large and often unserializable). The wrapper only holds a reference to a Configuration injected after deserialization; calling get() before that injection returns null and triggers this IllegalStateException.","triggerScenarios":"Calling NonSerializingConfig.get() when the underlying Configuration reference is null — i.e. code path that obtained the config via deserialization (e.g. a mapper/reducer or lambda serialized into a task) and no one called setConf/manually re-injected a Configuration before use.","commonSituations":"Shipping a closure or object referencing the input format's config into a MapReduce/Spark task without re-injecting the Hadoop Configuration; testing code that constructs NonSerializingConfig with a null conf.","solutions":["Set the Configuration manually after deserialization before calling get() (e.g. implement Configurable/setConf or re-inject via new NonSerializingConfig(conf)).","Do not capture the Configuration in objects that get serialized into distributed tasks; fetch it inside the task from the job's context (context.getConfiguration()).","If you own the code path, ensure the wrapper is initialized with a non-null Configuration before use."],"exampleFix":"// before\nConfiguration conf = serializedHolder.getConfig().get(); // throws IllegalStateException\n// after\nConfiguration conf = context.getConfiguration(); // re-inject in the task\nserializedHolder.setConfig(conf);\nConfiguration resolved = serializedHolder.getConfig().get();","handlingStrategy":"validation","validationCode":"if (configRef == null || configRef.get() == null) {\n  throw new IllegalArgumentException(\"Configuration must be set before use; inject via context.getConfiguration()\");\n}","typeGuard":"boolean hasConfig(NonSerializingConfig c) { return c != null && c.get() != null; } // guard in try/catch or via null-check wrapper if exposed","tryCatchPattern":"try {\n  Configuration conf = nonSerializingConfig.get();\n} catch (IllegalStateException e) {\n  conf = context.getConfiguration(); // re-inject and retry\n  nonSerializingConfig.set(conf);\n}","preventionTips":["Never capture Configuration-containing objects in closures serialized to tasks","Always re-obtain Configuration from the task context (context.getConfiguration()) inside the task","Implement Configurable on wrapper classes to auto-inject conf after serialization"],"tags":["hadoop","configuration","serialization","state-error"],"backgroundTag":"missing-required-config","analyzedSha":"86d9c8fc543e7c56c9f624eb725f76c9baff9570","analyzedAt":"2026-09-12T00:46:39.097Z","contentChangedAt":"2026-09-12T00:46:39.097Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}