What causes "cannot encode a null value" in apache/beam?

Fix the upstream code (e.g. DynamicDestinations.getTable) so it never returns null; return a valid TableDestination or fail with a descriptive error earlier.

apache/beam · error · CoderException

cannot encode a null value

Error message

cannot encode a null value

What it means

TableDestinationCoder.encode rejects null values before serialization. Beam custom coders are required to throw a CoderException when handed a null, because the wire format has no representation for null TableDestinations. It means a null TableDestination reached a shuffle/encode point that only accepts fully-formed destinations.

Solutions

Fix the upstream code (e.g. DynamicDestinations.getTable) so it never returns null; return a valid TableDestination or fail with a descriptive error earlier.
Filter out records that would produce a null destination before the BigQueryIO write transform.
In direct coder use, check value == null before calling encode and decide on explicit handling.

Example fix

// before
TableDestination dest = dynamicDestinations.getTable(row); // may be null
context.output(dest);
// after
TableDestination dest = dynamicDestinations.getTable(row);
if (dest == null) {
  throw new IllegalArgumentException("No destination resolved for record: " + row);
}
context.output(dest);

Defensive patterns

Strategy: validation

Validate before calling

if (dest == null) {
  throw new IllegalArgumentException("TableDestination must not be null before encode/write");
}

Type guard

boolean isValidDestination(TableDestination d) { return d != null && d.getTableSpec() != null; }

Prevention

Make DynamicDestinations.getTable total: never return null, throw a descriptive error instead.
Filter records that cannot resolve to a destination before the BigQueryIO write.

When it happens

Trigger: Calling TableDestinationCoder.INSTANCE.encode(null, outStream) directly, or a pipeline stage that emits/shuffles a null TableDestination (e.g. a DynamicDestinations returning null or a null key in a KV<TableDestination, ...> grouped write).

Common situations: Custom DynamicDestinations implementations that compute a table spec from records and return null when a field is missing; unit tests invoking encode with null to probe coder behavior.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/768aac704cac05b3. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoder.java:46

import org.checkerframework.checker.nullness.qual.Nullable;

/** A coder for {@link TableDestination} objects. */
public class TableDestinationCoder extends AtomicCoder<TableDestination> {
  private static final TableDestinationCoder INSTANCE = new TableDestinationCoder();
  private static final Coder<String> tableSpecCoder = StringUtf8Coder.of();
  private static final Coder<@Nullable String> tableDescriptionCoder =
      NullableCoder.of(StringUtf8Coder.of());

  private TableDestinationCoder() {}

  public static TableDestinationCoder of() {
    return INSTANCE;
  }

  @Override
  public void encode(TableDestination value, OutputStream outStream) throws IOException {
    if (value == null) {
      throw new CoderException("cannot encode a null value");
    }
    tableSpecCoder.encode(value.getTableSpec(), outStream);
    tableDescriptionCoder.encode(value.getTableDescription(), outStream);
  }

  @Override
  public TableDestination decode(InputStream inStream) throws IOException {
    String tableSpec = tableSpecCoder.decode(inStream);
    String tableDescription = tableDescriptionCoder.decode(inStream);
    return new TableDestination(tableSpec, tableDescription);
  }

  @Override
  public void verifyDeterministic() throws NonDeterministicException {}
}

View on GitHub (pinned to 12126d8942)