apache/beam · error · UnsupportedOperationException

%s is not a runner known to be compatible with Kafka exactly

Error message

%s is not a runner known to be compatible with Kafka exactly-once sink. This implementation of exactly-once sink relies on specific checkpoint guarantees. Only the runners with known to have compatible checkpoint semantics are allowed.

What it means

The Kafka exactly-once sink relies on runner checkpointing semantics to commit produced records atomically. Before expansion, KafkaIO verifies the configured runner is one with known-compatible checkpointing (Direct, Dataflow, Spark, Flink); any other runner raises this UnsupportedOperationException.

Source

Thrown at sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java:3597

                pCollectionTuple
                    .get(BadRecordRouter.BAD_RECORD_TAG)
                    .setCoder(BadRecord.getCoder(input.getPipeline())));
      }
      return PDone.in(input.getPipeline());
    }

    @Override
    public void validate(@Nullable PipelineOptions options) {
      Preconditions.checkStateNotNull(options);
      if (isEOS()) {
        String runner = options.getRunner().getName();
        if ("org.apache.beam.runners.direct.DirectRunner".equals(runner)
            || runner.startsWith("org.apache.beam.runners.dataflow.")
            || runner.startsWith("org.apache.beam.runners.spark.")
            || runner.startsWith("org.apache.beam.runners.flink.")) {
          return;
        }
        throw new UnsupportedOperationException(
            runner
                + " is not a runner known to be compatible with Kafka exactly-once sink. This"
                + " implementation of exactly-once sink relies on specific checkpoint guarantees."
                + " Only the runners with known to have compatible checkpoint semantics are"
                + " allowed.");
      }
    }

    // set config defaults
    private static final Map<String, Object> DEFAULT_PRODUCER_PROPERTIES =
        ImmutableMap.of(ProducerConfig.RETRIES_CONFIG, 3);

    /** A set of properties that are not required or don't make sense for our producer. */
    private static final Map<String, String> IGNORED_PRODUCER_PROPERTIES =
        ImmutableMap.of(
            ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, "Use withKeySerializer instead",
            ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, "Use withValueSerializer instead");

View on GitHub (pinned to 12126d8942)

Solutions

Run on a supported runner (Direct for tests, Dataflow, Spark, or Flink).
Disable exactly-once sink (drop withEOS / use ACK-based or at-least-once sink) if the runner can't change.
If you believe your runner has compatible checkpoint semantics, use a non-EOS sink or contact the Beam community; there is no override flag.

Example fix

// before
--runner=MyCustomRunner  (with .withEOS(30, "test"))
// after
--runner=FlinkRunner  (with .withEOS(30, "test"))

Defensive patterns

Strategy: validation

Validate before calling

String runner = options.getRunner().getName();
boolean eosCompatible = runner.equals("org.apache.beam.runners.direct.DirectRunner")
    || runner.startsWith("org.apache.beam.runners.dataflow.")
    || runner.startsWith("org.apache.beam.runners.spark.")
    || runner.startsWith("org.apache.beam.runners.flink.");
if (usingEosSink && !eosCompatible) {
  throw new IllegalArgumentException("Exactly-once Kafka sink requires Direct/Dataflow/Spark/Flink");
}

Try / catch

try {
  pipeline.run().waitUntilFinish();
} catch (UnsupportedOperationException e) {
  if (e.getMessage() != null && e.getMessage().contains("exactly-once sink")) {
    // reconfigure write without withEOS or switch runner
  }
  throw e;
}

Prevention

Only enable withEOS on Direct/Dataflow/Spark/Flink runners
Gate EOS configuration behind a pipeline-option check
For unsupported runners use the at-least-once sink with idempotent consumers

When it happens

Trigger: Using KafkaIO.writeRecords().withEOS(...) (exactly-once semantics) on a runner other than Direct/Dataflow/Spark/Flink — e.g. a custom or experimental runner, or a runner class name that doesn't start with the recognized package prefixes.

Common situations: Running exactly-once Kafka writes on a new/alternative runner (Samza, custom portable runner), or setting --runner to a subclass in an unrecognized package.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/55e6e220fd1fc951. Report an issue: GitHub.