apache/beam · error · java.lang.IllegalArgumentException

Cannot infer schema with a circular reference. Proto Field:

Error message

Cannot infer schema with a circular reference. Proto Field: ${name}

What it means

ProtoSchemaTranslator.getSchema() infers a Beam Schema from a protobuf Descriptor. It tracks descriptors currently being visited; if it re-enters a descriptor already on the stack (and it isn't the specially allowed Struct case, which has its own error), the proto message type is recursive and no finite schema can be inferred. The library throws IllegalArgumentException naming the offending proto field's full name.

Source

Thrown at sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java:186

  /** Return the proto field number for a type. */
  static int getFieldNumber(Field field) {
    return field.getOptions().getValue(SCHEMA_OPTION_META_NUMBER);
  }

  /** Return a Beam schema representing a proto class. */
  static Schema getSchema(Class<? extends Message> clazz) {
    return getSchema(ProtobufUtil.getDescriptorForClass(clazz));
  }

  static synchronized Schema getSchema(Descriptors.Descriptor descriptor) {
    if (alreadyVisitedSchemas.containsKey(descriptor)) {
      @Nullable Schema existingSchema = alreadyVisitedSchemas.get(descriptor);
      if (existingSchema == null) {
        String name = descriptor.getFullName();
        if ("google.protobuf.Struct".equals(name)) {
          throw new UnsupportedOperationException("Infer schema of Struct type is not supported.");
        }
        throw new IllegalArgumentException(
            "Cannot infer schema with a circular reference. Proto Field: " + name);
      }
      return existingSchema;
    }
    alreadyVisitedSchemas.put(descriptor, null);
    /* OneOfComponentFields refers to the field number in the protobuf where the component subfields
     * are. This is needed to prevent double inclusion of the component fields.*/
    Set<Integer> oneOfComponentFields = Sets.newHashSet();
    /* OneOfFieldLocation stores the field number of the first field in the OneOf. Using this, we can use the location
    of the first field in the OneOf as the location of the entire OneOf.*/
    Map<Integer, Field> oneOfFieldLocation = Maps.newHashMap();
    List<Field> fields = Lists.newArrayListWithCapacity(descriptor.getFields().size());

    // In proto3, an optional field is internally implemented by wrapping it in a synthetic oneof.
    // The Descriptor.getRealOneOfs() method is then used to retrieve only the "real" oneofs that
    // you explicitly defined, filtering out these automatically generated ones.
    // https://github.com/protocolbuffers/protobuf/blob/main/docs/implementing_proto3_presence.md#updating-a-
    for (OneofDescriptor oneofDescriptor : descriptor.getRealOneofs()) {

View on GitHub (pinned to 12126d8942)

Solutions

Break the recursion: wrap the self-referential field in google.protobuf.Struct or WellKnownTypes wrapper types, or convert that field to bytes/string and encode manually.
Flatten the recursive message into an iterative representation (e.g. a flat repeated list of nodes with parent ids) and infer the schema from that.
Provide the schema explicitly instead of inferring it (implement/construct the Beam Schema and coder by hand rather than using ProtoSchemaTranslator.getSchema).
If only Struct recursion is the issue, note google.protobuf.Struct is explicitly rejected too — redesign the payload to avoid Struct.

Example fix

// before
message Node {
  string name = 1;
  repeated Node children = 2; // circular reference -> IllegalArgumentException
}
// after
message Node {
  string name = 1;
  repeated string child_ids = 2; // flatten the tree into a sibling table
}

Defensive patterns

Strategy: validation

Validate before calling

// Java: reject recursive protos before calling getSchema
static boolean isRecursive(Descriptors.Descriptor d, Set<String> seen) {
  if (!seen.add(d.getFullName())) return true;
  for (Descriptors.FieldDescriptor f : d.getFields()) {
    if (f.getType() == Descriptors.FieldDescriptor.Type.MESSAGE
        && isRecursive(f.getMessageType(), new HashSet<>(seen))) return true;
  }
  return false;
}
// if (isRecursive(myMsg.getDescriptor(), new HashSet<>())) throw new IllegalArgumentException("recursive proto");

Try / catch

try { Schema s = ProtoSchemaTranslator.getSchema(descriptor); } catch (IllegalArgumentException e) { if (e.getMessage().startsWith("Cannot infer schema")) { /* redesign proto or supply schema manually */ } else throw e; }

Prevention

Design protos intended for Beam schema inference without cycles; prefer flattened repeated-entry representations for trees.
Add a unit test that infers schemas for every proto used in the pipeline so recursion is caught in CI.
Keep google.protobuf.Struct and self-referential types out of messages fed to ProtoSchemaTranslator.

When it happens

Trigger: Calling getSchema (directly or via protoToRow-based sources/transforms) on a proto message whose type graph contains a cycle, e.g. a message with a repeated field of its own type like `message Node { repeated Node children = 1; }`.

Common situations: Tree/graph-shaped protobuf definitions (linked lists, org charts, AST nodes); messages that mutually reference each other through two or more message types; reusing existing company protos designed for RPC, not schema inference.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/16016d7e4c0a7f39. Report an issue: GitHub.