apache/beam · error · IllegalArgumentException

Unable to parse schema {}

Error message

Unable to parse schema {}

What it means

JsonUtils.beamSchemaFromJsonSchema builds a Beam Schema from a JSON Schema. Beam requires a deterministic field ordering, so required properties are processed first from jsonSchema.getRequiredProperties(); if a name listed as required has no corresponding entry in the properties map, this IllegalArgumentException is thrown. The JSON Schema is inconsistent (required references an undefined property).

Source

Thrown at sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/JsonUtils.java:218

  public static Schema beamSchemaFromJsonSchema(String jsonSchemaStr) {
    org.everit.json.schema.ObjectSchema jsonSchema = jsonSchemaFromString(jsonSchemaStr);
    return beamSchemaFromJsonSchema(jsonSchema);
  }

  private static Schema beamSchemaFromJsonSchema(org.everit.json.schema.ObjectSchema jsonSchema) {
    Schema.Builder beamSchemaBuilder = Schema.builder();
    Map<String, org.everit.json.schema.Schema> properties =
        new HashMap<>(jsonSchema.getPropertySchemas());
    // Properties in a JSON Schema are stored in a Map object and unfortunately don't maintain
    // order. However, the schema's required properties is a list of property names that is
    // consistent and is in the same order as when the schema was first created. To create a
    // consistent Beam Schema from the same JSON schema, we add Schema Fields following this order.
    // We can guarantee a consistent Beam schema when all JSON properties are required.
    for (String propertyName : jsonSchema.getRequiredProperties()) {
      org.everit.json.schema.Schema propertySchema = properties.get(propertyName);
      if (propertySchema == null) {
        throw new IllegalArgumentException("Unable to parse schema " + jsonSchema);
      }

      Boolean isNullable =
          Boolean.TRUE.equals(propertySchema.getUnprocessedProperties().get("nullable"));
      beamSchemaBuilder =
          addPropertySchemaToBeamSchema(
              propertyName, propertySchema, beamSchemaBuilder, isNullable);
      // Remove properties we already added.
      properties.remove(propertyName, propertySchema);
    }

    // Now we are potentially left with properties that are not required. Add them too.
    // Note: having more than one non-required properties may result in  inconsistent
    // Beam schema field orderings.
    for (Map.Entry<String, org.everit.json.schema.Schema> entry : properties.entrySet()) {
      String propertyName = entry.getKey();
      org.everit.json.schema.Schema propertySchema = entry.getValue();

View on GitHub (pinned to 12126d8942)

Solutions

1. Remove the undefined name from the JSON Schema's 'required' array, or add a matching 'properties' entry
2. Validate the JSON Schema (required ⊆ properties) before passing it to Beam
3. Regenerate the schema from a single source so required and properties stay in sync

Example fix

// before
{"type":"object","required":["a","b"],"properties":{"a":{"type":"string"}}}
// after
{"type":"object","required":["a"],"properties":{"a":{"type":"string"}}}

static void validateRequiredSubset(JSONObject jsonSchema) {
  JSONObject props = jsonSchema.getJSONObject("properties");
  for (Object req : jsonSchema.optJSONArray("required")) {
    if (!props.has((String) req))
      throw new IllegalArgumentException("required property '" + req + "' missing from properties");
  }
}

static boolean requiredSubsetOfProperties(JSONObject schema) {
  JSONObject props = schema.optJSONObject("properties");
  if (props == null) return schema.optJSONArray("required") == null;
  java.util.stream.StreamSupport.stream(
      java.util.Spliterators.spliteratorUnknownSize(schema.optJSONArray("required")==null?java.util.Collections.emptyListIterator():schema.optJSONArray("required").iterator(),0),false)
      .allMatch(r -> props.has((String) r));
  return true;
}

try {
  Schema s = JsonUtils.beamSchemaFromJsonSchema(jsonSchema);
} catch (IllegalArgumentException e) {
  throw new SchemaParseException("Invalid JSON Schema (required vs properties mismatch): " + e.getMessage(), e);
}

When it happens

Trigger: Parsing a JSON Schema whose 'required' array contains a property name absent from 'properties' — via JsonUtils.beamSchemaFromJsonSchema / fromJsonSchema APIs.

Common situations: Hand-written or tool-generated JSON Schemas with a required field listed but never defined; schema evolution where a property was removed from 'properties' but not from 'required'.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Unsupported Beam to JSON type: {}
• Tuple-like arrays are unsupported. Expected a single item ty
• Unsupported field type {} in field {}
• Array schema is not properly formatted or unsupported ({}).
• Pipeline update will not be possible because the following t

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1b470122e13d8787. Report an issue: GitHub.