apache/beam · error · IllegalStateException

The current runner does not support SDF-based Kafka read pro

Error message

The current runner does not support SDF-based Kafka read properly and the replacement runner lacks the support for the following properties: %s. For example if you are using Dataflow then consider using Dataflow Runner v2.

What it means

When expanding KafkaIO.Read, Beam tries to replace the legacy read with an SDF-based implementation (ReadFromKafkaViaUnbounded). If the runner does not support SDF or cannot honor required properties (KafkaIOReadImplementationCompatibilityException), expansion fails. Dataflow v1 is the classic case; Runner v2 supports SDF.

Source

Thrown at sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java:1875

            new KafkaReadOverrideFactory<>());

    private static class KafkaReadOverrideFactory<K, V>
        implements PTransformOverrideFactory<
            PBegin, PCollection<KafkaRecord<K, V>>, ReadFromKafkaViaSDF<K, V>> {

      @Override
      public PTransformReplacement<PBegin, PCollection<KafkaRecord<K, V>>> getReplacementTransform(
          AppliedPTransform<PBegin, PCollection<KafkaRecord<K, V>>, ReadFromKafkaViaSDF<K, V>>
              transform) {
        try {
          return PTransformReplacement.of(
              transform.getPipeline().begin(),
              new ReadFromKafkaViaUnbounded<>(
                  transform.getTransform().kafkaRead,
                  transform.getTransform().keyCoder,
                  transform.getTransform().valueCoder));
        } catch (KafkaIOReadImplementationCompatibilityException e) {
          throw new IllegalStateException(
              "The current runner does not support SDF-based Kafka read properly "
                  + "and the replacement runner lacks the support for the following properties: "
                  + e.getConflictingProperties()
                  + ". For example if you are using Dataflow then consider using Dataflow Runner v2.");
        }
      }

      @Override
      public Map<PCollection<?>, ReplacementOutput> mapOutputs(
          Map<TupleTag<?>, PCollection<?>> outputs, PCollection<KafkaRecord<K, V>> newOutput) {
        return ReplacementOutputs.singleton(outputs, newOutput);
      }
    }

    private abstract static class AbstractReadFromKafka<K, V>
        extends PTransform<PBegin, PCollection<KafkaRecord<K, V>>> {
      Read<K, V> kafkaRead;
      Coder<K> keyCoder;

View on GitHub (pinned to 12126d8942)

Solutions

1. Switch to Dataflow Runner v2 (--experiments=beam_fnapi / use Runner v2 default) if using Dataflow.
2. Upgrade the runner to a version with full SDF support.
3. Run on DirectRunner/Flink/Spark which support the SDF-based read, for local testing.
4. Set --useDeprecatedKafkaRead (if available in your Beam version) to keep the legacy read implementation.

Example fix

// before (Dataflow v1)
--runner=DataflowRunner
// after
--runner=DataflowRunner --experiments=use_runner_v2

// Before submitting: verify runner supports SDF
if (options.getRunner() == DataflowRunner.class && !options.getExperiments().contains("use_runner_v2")) {
  options.setExperiments(ImmutableList.of("use_runner_v2"));
}

try {
  pipeline.run();
} catch (IllegalStateException e) {
  if (e.getMessage() != null && e.getMessage().contains("SDF-based Kafka read")) {
    throw new IllegalStateException("Re-run with Dataflow Runner v2: --experiments=use_runner_v2", e);
  }
  throw e;
}

When it happens

Trigger: Expanding KafkaIO.read() on a runner whose SDF support lacks properties reported by the compatibility check — e.g. Dataflow legacy runner, or a runner without watermark/track-property support — during pipeline construction.

Common situations: Running on old Dataflow (Runner v1), an outdated runner version without full SDF support, or a custom/portable runner missing required SDF capabilities.

Understand the failure class

Background: UnsupportedOperationException and "is not supported" errors: when a library deliberately refuses a call — this error's family across 30 libraries.

Related errors

• %s is not a runner known to be compatible with Kafka exactly
• There is no Kafka read implementation that supports every co
• Runner does not support draining.
• Class '%s' does not implement PipelineRunner. Supported pipe
• Unknown 'runner' specified '%s', supported pipeline runners

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/2adf25edade6197d. Report an issue: GitHub.