apache/beam · error · IllegalArgumentException

Firestore project id must be set on the transform or…

Error message

Firestore project id must be set on the transform or pipeline options.

What it means

The Firestore write SchemaTransform needs a GCP project id to address the database. It checks the transform configuration, then FirestoreOptions.getFirestoreProject(), then GcpOptions.getProject(); if all are unset/empty it throws this IllegalArgumentException. The pipeline cannot infer a project from the environment at graph-construction time.

Solutions

Set the pipeline option --project=<gcp-project-id> (or GOOGLE_CLOUD_PROJECT env var before submit)
Set --firestoreProject=<project-id> to specify the Firestore project explicitly
Pass projectId in the Firestore write configuration/transform builder if the API exposes it
For Dataflow templates, supply the project as a template parameter

Example fix

// before
pipeline.apply(FirestoreIO.v1().write().bulk().to("coll").build()); // no project anywhere
// after
// run with: --project=my-gcp-project  (or)
PipelineOptions opts = PipelineOptionsFactory.fromArgs(args).withValidation().create();
opts.as(FirestoreOptions.class).setFirestoreProject("my-gcp-project");

Defensive patterns

Strategy: validation

Validate before calling

GcpOptions gcp = options.as(GcpOptions.class); if (gcp.getProject() == null || gcp.getProject().isEmpty()) { throw new IllegalArgumentException("--project is required"); }

Try / catch

try { pipeline.apply(write); } catch (IllegalArgumentException e) { if (e.getMessage().contains("Firestore project id must be set")) { // surface a clear CLI/config error to the operator } throw e; }

Prevention

Always pass --project (or set GOOGLE_CLOUD_PROJECT) in CI and templates
Set FirestoreOptions.setFirestoreProject when multiple projects are in play
Fail fast with a config check at job submission rather than at graph construction

When it happens

Trigger: Expanding the Firestore write transform without setting the project id in configuration and without --firestoreProject or --project pipeline options; running where GcpOptions.getProject() was never populated (no --project flag, no GOOGLE_CLOUD_PROJECT resolution at submit time).

Common situations: Running pipelines on machines without gcloud-configured default project; CI environments lacking GOOGLE_CLOUD_PROJECT; template runs where the project parameter was omitted.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a4816082b91ed4cd. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreWriteSchemaTransformProvider.java:145

      PCollection<Row> errorOutput = outputTuple.get(ERROR_TAG).setRowSchema(errorSchema);
      ErrorHandling errorHandling = configuration.getErrorHandling();
      return PCollectionRowTuple.of(
          (handleErrors && errorHandling != null) ? errorHandling.getOutput() : "errors",
          errorOutput);
    }

    private String resolveProjectId(org.apache.beam.sdk.Pipeline pipeline) {
      if (!Strings.isNullOrEmpty(configuration.getProjectId())) {
        return configuration.getProjectId();
      }
      FirestoreOptions firestoreOptions = pipeline.getOptions().as(FirestoreOptions.class);
      if (!Strings.isNullOrEmpty(firestoreOptions.getFirestoreProject())) {
        return firestoreOptions.getFirestoreProject();
      }
      String project = pipeline.getOptions().as(GcpOptions.class).getProject();
      if (Strings.isNullOrEmpty(project)) {
        throw new IllegalArgumentException(
            "Firestore project id must be set on the transform or pipeline options.");
      }
      return project;
    }

    private String resolveDatabaseId(org.apache.beam.sdk.Pipeline pipeline) {
      if (!Strings.isNullOrEmpty(configuration.getDatabaseId())) {
        return configuration.getDatabaseId();
      }
      return pipeline.getOptions().as(FirestoreOptions.class).getFirestoreDb();
    }
  }

  static class RowToWriteFn extends DoFn<Row, Write> {
    private final Schema schema;
    private final String projectId;
    private final String databaseId;
    private final String collectionId;

View on GitHub (pinned to 12126d8942)