apache/beam · error · CoderException

unable to deserialize record

Error message

unable to deserialize record

What it means

WritableCoder decodes Hadoop Writable objects by reflectively instantiating the class and calling readFields. If instantiation or readFields fails via reflection, it throws this CoderException indicating the record could not be deserialized from the encoded stream.

Solutions

1. Give the Writable class a public no-arg constructor
2. Ensure the Writable class is on the worker classpath and in the same package/name at encode and decode time
3. Keep Writable field format stable across versions, or bump a serialVersionUID-like version marker and migrate data
4. Read the wrapped reflective exception for the exact failure (constructor vs readFields)

Example fix

// before
class MyWritable implements Writable {
  MyWritable(String s) { this.s = s; } // no no-arg ctor
}
// after
class MyWritable implements Writable {
  public MyWritable() {}
  MyWritable(String s) { this.s = s; }
}

Class<? extends Writable> c = writableClass;
c.getDeclaredConstructor().newInstance(); // fails fast if no accessible no-arg ctor

try {
  MyWritable w = coder.decode(stream, Coder.Context.OUTER);
} catch (CoderException e) {
  if (e.getMessage().contains("unable to deserialize record")) {
    LOGGER.severe("Writable decode failed: " + e.getCause());
  }
  throw e;
}

When it happens

Trigger: Decoding an encoded Writable (e.g. Text, IntWritable, custom Writable) when the class has no accessible no-arg constructor, the constructor throws, or readFields throws — often during pipeline deserialization of side inputs or grouped values.

Common situations: Custom Writable classes lacking a public no-arg constructor; class name changes between job submission and worker runtime; incompatible serialization format after upgrading Hadoop.

Related errors

• Hadoop Writable may be non-deterministic.
• cannot encode a null BitSet
• cannot encode a null byte[]
• cannot encode a null Count-min Sketch
• cannot encode a null Integer

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a43724f94800ab63. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/hadoop-common/src/main/java/org/apache/beam/sdk/io/hadoop/WritableCoder.java:93

    value.write(new DataOutputStream(outStream));
  }

  @SuppressWarnings("unchecked")
  @Override
  public T decode(InputStream inStream) throws IOException {
    try {
      if (type == NullWritable.class) {
        // NullWritable has no default constructor
        return (T) NullWritable.get();
      }
      T t = type.getDeclaredConstructor().newInstance();
      t.readFields(new DataInputStream(inStream));
      return t;
    } catch (InstantiationException
        | IllegalAccessException
        | NoSuchMethodException
        | InvocationTargetException e) {
      throw new CoderException("unable to deserialize record", e);
    }
  }

  @Override
  public List<Coder<?>> getCoderArguments() {
    return Collections.emptyList();
  }

  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(this, "Hadoop Writable may be non-deterministic.");
  }

  @Override
  public boolean equals(@Nullable Object other) {
    if (other == this) {
      return true;
    }

View on GitHub (pinned to 12126d8942)