apache/beam · error · IllegalArgumentException

Unknown policy ${policy}

Error message

Unknown policy ${policy}

What it means

Distinct.withNaturalKeys/apply logic selects a reduce strategy based on the configured TimestampPolicy. The getTransformFn switch handles OLDEST and ANY; any other policy value (e.g. NEWEST, or a policy enum extended in a newer version not handled here) falls through to default and throws IllegalArgumentException 'Unknown policy'.

Source

Thrown at sdks/java/extensions/euphoria/src/main/java/org/apache/beam/sdk/extensions/euphoria/core/client/operator/Distinct.java:384

      return MapElements.named(getName().orElse("") + "::extract-keys")
          .of(distinct)
          .using(KV::getKey, input.getTypeDescriptor())
          .output();
    }
    UnaryFunction<PCollection<InputT>, PCollection<InputT>> transformFn = getTransformFn();
    return transformFn.apply(input);
  }

  private UnaryFunction<PCollection<InputT>, PCollection<InputT>> getTransformFn() {
    switch (policy) {
      case NEWEST:
      case OLDEST:
        String name = getName().orElse(null);
        return input -> input.apply(TimestampExtractTransform.of(name, this::reduceTimestamped));
      case ANY:
        return this::reduceSelectingAny;
      default:
        throw new IllegalArgumentException("Unknown policy " + policy);
    }
  }

  private PCollection<InputT> reduceSelectingAny(PCollection<InputT> input) {
    return ReduceByKey.named(getName().orElse(null))
        .of(input)
        .keyBy(getKeyExtractor(), getKeyType().orElse(null))
        .valueBy(e -> e, getOutputType().orElse(null))
        .combineBy(values -> nonEmpty(values.findAny()), getOutputType().orElse(null))
        .outputValues();
  }

  private PCollection<InputT> reduceTimestamped(PCollection<KV<Long, InputT>> input) {
    CombinableReduceFunction<KV<Long, InputT>> select = getReduceFn();
    PCollection<KV<Long, InputT>> outputValues =
        ReduceByKey.named(getName().orElse(null))
            .of(input)
            .keyBy(e -> getKeyExtractor().apply(e.getValue()), getKeyType().orElse(null))

View on GitHub (pinned to 12126d8942)

Solutions

1. Use only TimestampPolicy.OLDEST or TimestampPolicy.ANY with this operator.
2. If NEWEST-like behavior is needed, implement a custom ReduceByKey pipeline instead of relying on Distinct.
3. Check the Beam version: if a newer version handles the policy, upgrade the org.apache.beam euphoria extension dependency.

Example fix

// before
Distinct.named("d").withTimestampPolicy(Distinct.TimestampPolicy.NEWEST);
// after
Distinct.named("d").withTimestampPolicy(Distinct.TimestampPolicy.OLDEST);

if (policy != Distinct.TimestampPolicy.OLDEST && policy != Distinct.TimestampPolicy.ANY) {
  throw new IllegalArgumentException("Unsupported policy for Distinct: " + policy);
}

try {
  expansion = getTransformFnFor(policy);
} catch (IllegalArgumentException e) {
  LOG.warn("falling back to TimestampPolicy.ANY");
  expansion = getTransformFnFor(Distinct.TimestampPolicy.ANY);
}

When it happens

Trigger: Building a Distinct operator with Distinct.TimestampPolicy set to a value not covered by the switch (typically NEWEST or an enum value added after this code was written), then triggering the operator's expansion via transformFn.

Common situations: Copying example code that references a policy variant supported elsewhere but not in this euphoria extension; upgrading the Beam version so the enum gained a new constant while the switch wasn't updated.

Understand the failure class

Background: Invalid enum value errors: "Unknown type", "Invalid scope", "must be one of" — when a string is not on the library's allowed list — this error's family across 23 libraries.

Related errors

• Unknown accumulation mode [${accumulationMode}]
• Unknown failure mode + failureMode
• Unrecognized value for stable unique names:
• Unknown ValueKind number: {}
• Unsupported compression type: " + compression

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/01b6ce833fa75fc5. Report an issue: GitHub.