apache/beam · error · IllegalArgumentException

Firestore project id must be set on the transform or…

Error message

Firestore project id must be set on the transform or pipeline options.

What it means

The Firestore read transform needs a GCP project id to target. resolveProjectId checks configuration, then FirestoreOptions.getFirestoreProject(), then GcpOptions.getProject(); if all are null/empty it throws IllegalArgumentException because it cannot construct Firestore RPCs without a project.

Solutions

Set projectId in the Firestore read SchemaTransform configuration
Pass --project=<gcp-project-id> on the pipeline command line
Set FirestoreOptions: --firestoreProject=<project> (or setFirestoreProject in code)
Verify GcpOptions.getProject() is populated in the runner environment (e.g. Dataflow usually sets it; DirectRunner may not)

Example fix

// before
Pipeline p = Pipeline.create(); // no --project, no projectId in config
// after
Pipeline p = Pipeline.create(PipelineOptionsFactory.fromArgs(new String[]{"--project=my-gcp-project"}).create());

Defensive patterns

Strategy: validation

Validate before calling

String project = options.as(GcpOptions.class).getProject();
if (project == null || project.isEmpty()) {
  throw new IllegalArgumentException("--project must be set for Firestore reads");
}

Try / catch

try {
  pipeline.apply(firestoreReadTransform);
} catch (IllegalArgumentException e) {
  LOG.error("Set projectId in config or --project/--firestoreProject options", e);
}

Prevention

Always pass --project on pipeline launch (works on DirectRunner too)
Or set projectId explicitly in the SchemaTransform configuration
For Firestore-specific override, set --firestoreProject

When it happens

Trigger: Creating the read SchemaTransform without a projectId in its configuration while the pipeline options lack both --firestoreProject (or FirestoreOptions.firestoreProject) and --project (GcpOptions.project).

Common situations: Running pipelines with only --tempLocation set; local/DirectRunner test runs without GCP options configured; provider config missing the project field; environments relying on ADC that don't pass project via options.

Understand the failure class

Background: "is required", "must be set", "missing required field": configuration validation errors across open-source libraries — this error's family across 36 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/8ef5a8812c489f7c. Report an issue: GitHub.

Appendix: source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreReadSchemaTransformProvider.java:165

        output =
            output.and(
                configuration.getErrorHandling().getOutput(),
                outputTuple.get(ERROR_TAG).setRowSchema(errorSchema));
      }
      return output;
    }

    private String resolveProjectId(org.apache.beam.sdk.Pipeline pipeline) {
      if (!Strings.isNullOrEmpty(configuration.getProjectId())) {
        return configuration.getProjectId();
      }
      FirestoreOptions firestoreOptions = pipeline.getOptions().as(FirestoreOptions.class);
      if (!Strings.isNullOrEmpty(firestoreOptions.getFirestoreProject())) {
        return firestoreOptions.getFirestoreProject();
      }
      String project = pipeline.getOptions().as(GcpOptions.class).getProject();
      if (Strings.isNullOrEmpty(project)) {
        throw new IllegalArgumentException(
            "Firestore project id must be set on the transform or pipeline options.");
      }
      return project;
    }

    private String resolveDatabaseId(org.apache.beam.sdk.Pipeline pipeline) {
      if (!Strings.isNullOrEmpty(configuration.getDatabaseId())) {
        return configuration.getDatabaseId();
      }
      return pipeline.getOptions().as(FirestoreOptions.class).getFirestoreDb();
    }
  }

  static class DocumentToRowFn extends DoFn<Document, Row> {
    private final Schema schema;
    private final @org.checkerframework.checker.nullness.qual.Nullable String documentIdField;
    private final boolean handleErrors;
    private final Schema errorSchema;

View on GitHub (pinned to 12126d8942)