apache/beam · error · NonDeterministicException

TableRow can hold arbitrary instances, which may be…

Error message

TableRow can hold arbitrary instances, which may be non-deterministic.

What it means

BigQueryInsertErrorCoder.verifyDeterministic() always throws NonDeterministicException. A TableRow can hold arbitrary Object instances (e.g. Maps, custom objects) whose encoding may differ run to run, so the coder cannot guarantee deterministic encoding — a requirement for operations like grouping or stateful processing.

Solutions

1. Extract deterministic fields (e.g. row content as JSON string, error message, table destination) into a stable POJO/String and group on that instead.
2. Apply a MapElements/ParDo to convert BigQueryInsertError to a deterministically-coded type before GroupByKey.
3. If only counting/metrics are needed, transform to simple key-value pairs (String/Long) before grouping.

Example fix

// before
failedInserts.apply(GroupByKey.create()); // BigQueryInsertError is non-deterministic
// after
failedInserts
    .apply(MapElements.via(new SimpleFunction<BigQueryInsertError, KV<String, String>>() {
      public KV<String, String> apply(BigQueryInsertError e) {
        return KV.of(e.getTableRow().get("k").toString(), e.getErrorMessages().toString());
      }
    }))
    .apply(GroupByKey.create());

if (willBeGrouped(pcOfInsertErrors)) {
  throw new IllegalStateException("BigQueryInsertError is non-deterministically coded; map to a deterministic type before grouping");
}

static boolean isDeterministicallyCodable(Class<?> c) {
  return !BigQueryInsertError.class.isAssignableFrom(c);
}

try {
  pipeline.apply(GroupByKey.create());
} catch (NonDeterministicException e) {
  if (e.getMessage() != null && e.getMessage().contains("TableRow can hold arbitrary instances")) {
    throw new IllegalStateException("Project BigQueryInsertError onto a deterministic key/value type first", e);
  } else { throw e; }
}

When it happens

Trigger: Using BigQueryInsertError (e.g. from BigQueryIO write results with failed inserts) in an operation that requires deterministic coding, such as GroupByKey, Combine, or CoGroupByKey; the coder's verifyDeterministic is invoked during pipeline validation.

Common situations: Pipelines that join or aggregate failed-insert records to build retry batches or dead-letter metrics; developers keying on BigQueryInsertError or wrapping it in keyed PCollections.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• A function must be provided to convert the input type into…
• ApproximateUnique.PerKey requires its input to use KvCoder
• AvroCoder for GenericRecord requires a schema
• BigQueryIO.Write transforms cannot be converted to a…
• Both numFileShards and auto-sharding options are set. Will…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/711d327cbe1023c7. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryInsertErrorCoder.java:91

  // properly deserialized as such and not as an Integer instead.
  private static final ObjectMapper MAPPER =
      new ObjectMapper()
          .disable(SerializationFeature.FAIL_ON_EMPTY_BEANS)
          .enableDefaultTyping(ObjectMapper.DefaultTyping.NON_FINAL);

  private static final BigQueryInsertErrorCoder INSTANCE = new BigQueryInsertErrorCoder();
  private static final TypeDescriptor<BigQueryInsertError> TYPE_DESCRIPTOR =
      new TypeDescriptor<BigQueryInsertError>() {};

  /**
   * {@inheritDoc}
   *
   * @throws NonDeterministicException always. A {@link TableRow} can hold arbitrary {@link Object}
   *     instances, which makes the encoding non-deterministic.
   */
  @Override
  public void verifyDeterministic() throws NonDeterministicException {
    throw new NonDeterministicException(
        this, "TableRow can hold arbitrary instances, which may be non-deterministic.");
  }

  @Override
  public TypeDescriptor<BigQueryInsertError> getEncodedTypeDescriptor() {
    return TYPE_DESCRIPTOR;
  }
}

View on GitHub (pinned to 12126d8942)