What causes "Record Writer not set for FilterRecordWriter" in apache/hadoop?

Assign rawWriter in the constructor of your FilterRecordWriter subclass

apache/hadoop · error · IOException

Record Writer not set for FilterRecordWriter

Error message

Record Writer not set for FilterRecordWriter

What it means

FilterRecordWriter is a convenience RecordWriter wrapper whose write() and close() delegate to the RecordWriter stored in its protected rawWriter field. getRawWriter() throws this IOException when write() or close() is called before a subclass assigned rawWriter. It indicates the wrapper was created but never given the underlying writer it forwards to.

Source

Thrown at hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/lib/FilterOutputFormat.java:97

    public FilterRecordWriter() throws IOException {
      rawWriter = null;
    }

    public FilterRecordWriter(RecordWriter<K,V> rawWriter)  throws IOException {
      this.rawWriter = rawWriter;
    }

    public void close(Reporter reporter) throws IOException {
      getRawWriter().close(reporter);
    }

    public void write(K key, V value) throws IOException {
      getRawWriter().write(key, value);
    }
    
    private RecordWriter<K,V> getRawWriter() throws IOException {
      if (rawWriter == null) {
        throw new IOException ("Record Writer not set for FilterRecordWriter");
      }
      return rawWriter;
    }
  }

}

View on GitHub (pinned to 2add963021)

Solutions

Assign rawWriter in the constructor of your FilterRecordWriter subclass
If rawWriter is created lazily, override close() to guard: if (rawWriter != null) rawWriter.close(reporter);
Verify every code path that can call write()/close() has initialized the delegate first

Example fix

// before: close() NPE-path -> IOException "Record Writer not set for FilterRecordWriter"
public void close(Reporter reporter) throws IOException {
  getRawWriter().close(reporter); // rawWriter null when nothing was written
}

// after: guard the delegate
public void close(Reporter reporter) throws IOException {
  if (rawWriter != null) {
    rawWriter.close(reporter);
  }
}

Defensive patterns

Strategy: validation

Validate before calling

// guard every delegation in a FilterRecordWriter subclass
public void close(Reporter reporter) throws IOException {
  if (rawWriter != null) {
    rawWriter.close(reporter);
  }
}
public void write(K key, V value) throws IOException {
  if (rawWriter == null) throw new IOException("writer not initialized for this path");
  rawWriter.write(key, value);
}

Try / catch

try {
  writer.close(reporter);
} catch (IOException e) {
  if (e.getMessage().contains("not set for FilterRecordWriter")) {
    LOG.warn("no underlying writer; nothing was written through this wrapper");
    return;
  }
  throw e;
}

Prevention

Initialize rawWriter in the constructor whenever possible
For lazily-created writers, always pair the lazy creation with a null guard in close()
Cover the zero-records path in unit tests — it is exactly where close() hits a null delegate

When it happens

Trigger: Extending FilterRecordWriter (the inner class of FilterOutputFormat) without assigning rawWriter, then the framework calls write() during the map/reduce phase or close() at task commit. A subtler variant: a subclass that creates rawWriter lazily on first write(), but the task writes zero records so close() runs first and hits the null rawWriter.

Common situations: Lazy-writer patterns (like LazyOutputFormat's LazyRecordWriter) copied without the null guard in close(); wrapper writers that conditionally create the underlying writer and skip creation when no data was written.

Related errors

AI-assisted analysis of apache/hadoop@2add963021 (2026-08-22). Data as JSON: /api/errors/bd650ce473112499. Report an issue: GitHub.