Could not use the provided `BigQueryServices`…

Error message

Could not use the provided `BigQueryServices` implementation when upgrading.Using the default.

What it means

When Beam deserializes a persisted BigQueryIO write transform from its config Row (e.g. during pipeline translation or an SDK upgrade), it tries to restore a serialized custom BigQueryServices implementation. If deserialization throws InvalidClassException (class incompatible with the serialized version), Beam logs this warning and falls back to the default BigQueryServicesImpl.

Solutions

Re-generate the pipeline with the current SDK version instead of replaying old serialized config rows.
Implement the custom BigQueryServices with a fixed serialVersionUID matching the serialized version, or ensure class compatibility across versions.
Accept the fallback if default BigQueryServicesImpl behavior is acceptable — this is only a warning.
Remove the custom setBigQueryServices if the default implementation suffices, avoiding the serialized class entirely.

Example fix

// before
public class MyBqServices implements BigQueryServices { /* no serialVersionUID */ }
// after
public class MyBqServices implements BigQueryServices {
  private static final long serialVersionUID = 1L;
}

Defensive patterns

Strategy: fallback

Validate before calling

// Before relying on persisted custom services, verify deserializability:
try { fromByteArray(bigqueryServicesBytes); } catch (InvalidClassException e) { /* plan fallback */ }

Type guard

boolean isDeserializable(byte[] b) { try { fromByteArray(b); return true; } catch (InvalidClassException e) { return false; } }

Try / catch

try { builder.setBigQueryServices((BigQueryServices) fromByteArray(bytes)); } catch (InvalidClassException e) { builder.setBigQueryServices(new BigQueryServicesImpl()); }

Prevention

Declare serialVersionUID in custom BigQueryServices classes.
Deserialize pipelines with the same SDK version that serialized them.
Keep default BigQueryServices as a tested fallback path.

When it happens

Trigger: Reading a BigQueryIO write transform from a config Row (readTransformFromRow -> fromConfigRow) where 'bigquery_services' bytes were serialized by a different Beam/SDK version, producing an InvalidClassException during fromByteArray.

Common situations: Upgrading the Beam SDK between pipeline versions (e.g. updating runner images mid-pipeline); running an old serialized pipeline snapshot with a newer SDK; custom BigQueryServices class serialVersionUID changed between releases.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/887990a95fda1419. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIOTranslation.java:290

        if (flattenResults != null) {
          builder = builder.setFlattenResults(flattenResults);
        }
        Boolean useLegacySQL = configRow.getBoolean("use_legacy_sql");
        if (useLegacySQL != null) {
          builder = builder.setUseLegacySql(useLegacySQL);
        }
        Boolean withTemplateCompatibility = configRow.getBoolean("with_template_compatibility");
        if (withTemplateCompatibility != null) {
          builder = builder.setWithTemplateCompatibility(withTemplateCompatibility);
        }
        byte[] bigqueryServicesBytes = configRow.getBytes("bigquery_services");
        if (bigqueryServicesBytes != null) {
          try {
            builder =
                builder.setBigQueryServices(
                    (BigQueryServices) fromByteArray(bigqueryServicesBytes));
          } catch (InvalidClassException e) {
            LOG.warn(
                "Could not use the provided `BigQueryServices` implementation when upgrading."
                    + "Using the default.");
            builder.setBigQueryServices(new BigQueryServicesImpl());
          }
        }
        byte[] parseFnBytes = configRow.getBytes("parse_fn");
        if (parseFnBytes != null) {
          builder = builder.setParseFn((SerializableFunction) fromByteArray(parseFnBytes));
        }
        byte[] datumReaderFactoryBytes = configRow.getBytes("datum_reader_factory");
        if (datumReaderFactoryBytes != null) {
          builder =
              builder.setDatumReaderFactory(
                  (SerializableFunction) fromByteArray(datumReaderFactoryBytes));
        }
        byte[] queryPriorityBytes = configRow.getBytes("query_priority");
        if (queryPriorityBytes != null) {
          builder = builder.setQueryPriority((QueryPriority) fromByteArray(queryPriorityBytes));

View on GitHub (pinned to 12126d8942)