apache/beam · error · IllegalArgumentException

It is required to set useCursorFetch=true in the JDBC URL wh

Error message

It is required to set useCursorFetch=true in the JDBC URL when using fetchSize for MySQL

What it means

This IllegalArgumentException is thrown by the MySQL JDBC read provider's from() configuration method in ReadFromMySqlSchemaTransformProvider. Beam's MySQL reader requires the MySQL JDBC driver's cursor-based fetch mode to actually honor a small fetchSize; without useCursorFetch=true in the JDBC URL, the driver buffers the entire result set in memory regardless of fetchSize, so the provider refuses the configuration. It is a fail-fast validation to prevent silent full-result-set materialization and OOM.

Source

Thrown at sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/providers/ReadFromMySqlSchemaTransformProvider.java:73

  @Override
  public @UnknownKeyFor @NonNull @Initialized SchemaTransform from(
      JdbcReadSchemaTransformConfiguration configuration) {
    String jdbcType = configuration.getJdbcType();
    if (jdbcType != null && !jdbcType.isEmpty() && !jdbcType.equals(jdbcType())) {
      LOG.warn(
          "Wrong JDBC type. Expected '{}' but got '{}'. Overriding with '{}'.",
          jdbcType(),
          jdbcType,
          jdbcType());
      configuration = configuration.toBuilder().setJdbcType(jdbcType()).build();
    }

    Integer fetchSize = configuration.getFetchSize();
    if (fetchSize != null
        && fetchSize > 0
        && configuration.getJdbcUrl() != null
        && !configuration.getJdbcUrl().contains("useCursorFetch=true")) {
      throw new IllegalArgumentException(
          "It is required to set useCursorFetch=true"
              + " in the JDBC URL when using fetchSize for MySQL");
    }
    return new MySqlReadSchemaTransform(configuration);
  }

  public static class MySqlReadSchemaTransform extends JdbcReadSchemaTransform {
    public MySqlReadSchemaTransform(JdbcReadSchemaTransformConfiguration config) {
      super(config, MYSQL);
    }
  }
}

View on GitHub (pinned to 12126d8942)

Solutions

Append useCursorFetch=true to the JDBC URL, e.g. jdbc:mysql://host:3306/db?useCursorFetch=true
If you do not need bounded fetching, remove the fetchSize setting from the configuration so the check passes
Ensure the URL is passed with exact spelling useCursorFetch=true (case-sensitive substring check) and that it is the URL actually supplied to the provider, not another config layer

Example fix

// before
JdbcReadSchemaTransformConfiguration cfg = JdbcReadSchemaTransformConfiguration.builder()
    .setJdbcUrl("jdbc:mysql://host:3306/db")
    .setFetchSize(1000)
    .build();
// after
JdbcReadSchemaTransformConfiguration cfg = JdbcReadSchemaTransformConfiguration.builder()
    .setJdbcUrl("jdbc:mysql://host:3306/db?useCursorFetch=true")
    .setFetchSize(1000)
    .build();

Defensive patterns

Strategy: validation

Validate before calling

if (fetchSize != null && fetchSize > 0 && jdbcUrl != null && !jdbcUrl.contains("useCursorFetch=true")) {
  throw new IllegalArgumentException("MySQL read requires useCursorFetch=true in the JDBC URL when fetchSize is set");
}

Prevention

Always include useCursorFetch=true in MySQL JDBC URLs used with Beam
Centralize URL construction in one helper so required params are never missing
When setting fetchSize, review provider-specific JDBC URL requirements per dialect

When it happens

Trigger: Calling ReadFromMySqlSchemaTransformProvider.from() (or its builder read()) with a configuration where fetchSize is set to a positive integer while the jdbcUrl does not contain the substring 'useCursorFetch=true'.

Common situations: Developers copy a working Postgres/SQL Server JDBC URL into the MySQL read config and then set fetchSize to bound memory usage; or they set fetchSize in a pipeline sent to a Dataflow/expansion service without updating the URL parameters. Also common after upgrading Beam where fetchSize support for MySQL was added with this requirement.

Understand the failure class

Background: "Invalid value" and "allowed values are" config errors: what your library rejected and how to fix it — this error's family across 41 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/5d16be2ee4428fdf. Report an issue: GitHub.