apache/beam · error · ValidationException

ValidationException

Error message

ValidationException

What it means

FailureCollectorWrapper.getOrThrowException aggregates validation failures collected during CDAP pipeline translation; once checks (such as removing schema-validation failures) are done, any remaining failures cause a ValidationException carrying the whole failure collection to be thrown. This is the standard CDAP failure-collector pattern: failures are gathered during validation and thrown together at a checkpoint.

Solutions

1. Inspect the ValidationException's failure collection; each entry names the offending field/property and reason
2. Fix the schema/config mismatch reported (align field names and types between the Beam schema and plugin config)
3. If the failure is a schema-field-only validation you intentionally want to ignore, adjust config so it passes instead of suppressing
4. Run with validation logging enabled to see all failures before the throw

Example fix

// before
schema has field "user_name" but plugin config references "username"
// after
align plugin config field "username" to schema field "user_name" (or vice versa) so no validation failures remain

// before getOrThrowException
FailureCollectorWrapper collector = ...;
if (!collector.getValidationFailures().isEmpty()) {
  collector.getValidationFailures().forEach(f -> LOG.warn("Validation failure: {}", f));
}

try {
  expand(...);
} catch (ValidationException e) {
  e.getFailures().forEach(f -> LOG.error("CDAP validation: {}", f.getMessage()));
  throw new IllegalArgumentException("Invalid CDAP plugin config/schema: see logged failures", e);
}

When it happens

Trigger: Any addFailure(...) on the wrapped collector followed by getOrThrowException with outstanding failures — e.g., schema/field validation problems detected while expanding a CDAP source/sink (missing fields, type mismatches between the schema and the plugin config).

Common situations: Mismatch between the declared schema and actual plugin config (renamed or removed fields); invalid option values surfaced during pipeline validation; running a pipeline where a CDAP plugin reports configuration errors at expansion time.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Given message schema
• Given message schema
• Provided length is bigger than max length
• Provided size is bigger than max size
• Received an empty Map, but output schema contains required…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/786c8de2e125e103. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/cdap/src/main/java/org/apache/beam/sdk/io/cdap/context/FailureCollectorWrapper.java:66

    // anything in our case
    List<ValidationFailure> schemaValidationFailures = new ArrayList<>();
    for (ValidationFailure failure : failuresCollection) {
      List<ValidationFailure.Cause> causes = failure.getCauses();
      if (causes != null) {
        for (ValidationFailure.Cause cause : causes) {
          String inputField = cause.getAttribute(CauseAttributes.INPUT_SCHEMA_FIELD);
          if (BatchContextImpl.DEFAULT_SCHEMA_FIELD_NAME.equals(inputField)) {
            schemaValidationFailures.add(failure);
          }
        }
      }
    }
    failuresCollection.removeAll(schemaValidationFailures);
    if (failuresCollection.isEmpty()) {
      return new ValidationException(this.failuresCollection);
    }

    throw new ValidationException(this.failuresCollection);
  }

  @Override
  public List<ValidationFailure> getValidationFailures() {
    return this.failuresCollection;
  }
}

View on GitHub (pinned to 12126d8942)