apache/beam · error · CoderException

cannot encode a null ByteString

Error message

cannot encode a null ByteString

What it means

ByteStringCoder.encode() rejects null values because a Coder in Beam must produce bytes for every element and nulls have no defined encoding. Throwing CoderException early surfaces the problem at encode time instead of writing ambiguous bytes. Nulls are not representable in Beam PCollections for coded types.

Solutions

1. Filter nulls before the coder sees them: apply Filter.by(v -> v != null).
2. Replace null emission with ByteString.EMPTY in the producing DoFn.
3. Wrap in an Optional-like type or a nullable wrapper if null is meaningful, with a coder that handles the wrapper.
4. Log/fix the upstream source that produced the null element.

Example fix

// before
context.output(value == null ? null : value.toStringUtf8().length() > 0 ? value : null);
// after
if (value != null) {
  c.output(value);
}

// before writing/encoding
if (elements.apply(Filter.by(v -> v == null)).getCount().read() > 0) {
  throw new IllegalStateException("null ByteStrings present; filter or replace with ByteString.EMPTY");
}

static boolean isEncodable(ByteString v) { return v != null; }

try {
  coder.encode(value, out, context);
} catch (CoderException e) {
  if (e.getMessage() != null && e.getMessage().contains("null ByteString")) {
    coder.encode(ByteString.EMPTY, out, context); // or skip element
  } else { throw e; }
}

When it happens

Trigger: A PCollection<ByteString> contains a null element and the element is encoded (e.g. written to a sink or shuffled); typically produced by a DoFn that emits null, or an I/O source producing null values.

Common situations: ParDo returning null instead of skipping; joins producing null for absent keys then feeding a protobuf/ByteString coder; GroupByKey on KV<ByteString, ...> where a null ByteString was emitted.

Related errors

• cannot encode a null BitSet
• cannot encode a null byte[]
• cannot encode a null Integer
• cannot encode a null Long
• cannot encode a null Short

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/47532e73fdf276bc. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ByteStringCoder.java:61

  /** ************************ */
  private static final ByteStringCoder INSTANCE = new ByteStringCoder();

  private static final TypeDescriptor<ByteString> TYPE_DESCRIPTOR =
      new TypeDescriptor<ByteString>() {};

  private ByteStringCoder() {}

  @Override
  public void encode(ByteString value, OutputStream outStream) throws IOException, CoderException {
    encode(value, outStream, Context.NESTED);
  }

  @Override
  public void encode(ByteString value, OutputStream outStream, Context context)
      throws IOException, CoderException {
    if (value == null) {
      throw new CoderException("cannot encode a null ByteString");
    }

    if (!context.isWholeStream) {
      // ByteString is not delimited, so write its size before its contents.
      VarInt.encode(value.size(), outStream);
    }
    value.writeTo(outStream);
  }

  @Override
  public ByteString decode(InputStream inStream) throws IOException {
    return decode(inStream, Context.NESTED);
  }

  @Override
  public ByteString decode(InputStream inStream, Context context) throws IOException {
    if (context.isWholeStream) {
      return ByteString.readFrom(inStream);

View on GitHub (pinned to 12126d8942)