apache/beam · error · RuntimeException

Length of Schema.Field[${field.getName()}] data exceeds data

Error message

Length of Schema.Field[${field.getName()}] data exceeds database column capacity

What it means

Pre-write validation in JdbcUtil that checks whether a row field's data length exceeds the declared maximum capacity (from the field's logical type argument, e.g. VARCHAR/CHAR/DECIMAL max length). It throws a RuntimeException before any SQL is executed when length > maxLimit, preventing a driver-level overflow error.

Source

Thrown at sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java:423

      }
      ps.setString(i + 1, value);
    };
  }

  private static void validateLogicalTypeLength(Schema.Field field, Integer length) {
    try {
      if (!field.getType().getTypeName().isLogicalType()) {
        return;
      }

      Integer maxLimit =
          (Integer) checkArgumentNotNull(field.getType().getLogicalType()).getArgument();
      if (maxLimit == null) {
        return;
      }

      if (length > maxLimit) {
        throw new RuntimeException(
            String.format(
                "Length of Schema.Field[%s] data exceeds database column capacity",
                field.getName()));
      }
    } catch (NumberFormatException e) {
      // if argument is not set or not integer then do nothing and proceed with the insertion
    }
  }

  private static Calendar getDateOrTimeOnly(DateTime dateTime, boolean wantDateOnly) {
    Calendar cal = Calendar.getInstance();
    cal.setTimeZone(TimeZone.getTimeZone(dateTime.getZone().getID()));

    if (wantDateOnly) { // return date only
      cal.set(Calendar.YEAR, dateTime.getYear());
      cal.set(Calendar.MONTH, dateTime.getMonthOfYear() - 1);
      cal.set(Calendar.DATE, dateTime.getDayOfMonth());

View on GitHub (pinned to 12126d8942)

Solutions

Increase the logical type argument (e.g. VARCHAR length) in the schema to cover the actual data
Truncate or validate the data upstream before the JdbcIO write
Remove the length argument from the logical type if the DB column is unlimited (validation is skipped when maxLimit is null)
Align the Beam schema with the real table DDL via SchemaUtil.toBeamSchema on read side

Example fix

// before
FieldType t = FieldType.STRING.withLogicalType(Records.sqlTypeToLogicalType("VARCHAR", 10));
// after
FieldType t = FieldType.STRING.withLogicalType(Records.sqlTypeToLogicalType("VARCHAR", 255));

Defensive patterns

Strategy: validation

Validate before calling

boolean validateLengths(Row row, Schema schema) {
  for (int i = 0; i < schema.getFieldCount(); i++) {
    Schema.Field f = schema.getField(i);
    if (f.getType().getLogicalType() == null) continue;
    Object arg = f.getType().getLogicalType().getArgument();
    if (!(arg instanceof Integer)) continue;
    int max = (Integer) arg;
    Object v = row.getValue(i);
    if (v instanceof String && ((String) v).length() > max) return false;
  }
  return true;
}

Try / catch

try {
  rows.apply(JdbcIO.<Row>write()...);
} catch (RuntimeException e) {
  if (e.getMessage().contains("exceeds database column capacity")) {
    LOG.error("Row value longer than schema logical-type limit");
  }
  throw e;
}

Prevention

Generate the Beam schema from the database via SchemaUtil.toBeamSchema rather than hand-writing it
Keep logical-type length arguments at least as large as the widest expected value
Add a length-checking ParDo before the write for early failure

When it happens

Trigger: Using JdbcIo.write with a schema whose field has a logical type with an integer argument (max length, e.g. VARCHAR(n) or DECIMAL(p,s)) and supplying a value whose length (string chars or numeric precision) exceeds that argument.

Common situations: User declares VARCHAR(10) in the Beam schema but feeds strings of 50 chars; mapping an existing Avro/Beam schema with smaller limits than the data; after someone lowered the logical-type argument in code.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/615d9c3ea2b72836. Report an issue: GitHub.