table does not exist and create disposition is 'CreateNever'

Error message

table does not exist and create disposition is 'CreateNever': %v

What it means

When writing to BigQuery with a CreateDisposition of CreateNever, Beam checks whether the destination table exists. If the table metadata lookup returns NotFound and creation is disallowed, this error is thrown because the write cannot proceed.

Source

Thrown at sdks/go/pkg/beam/io/bigqueryio/bigquery.go:363

		return err
	}
	defer client.Close()

	// TODO(herohde) 7/14/2017: should we create datasets? For now, "no".

	dataset := client.DatasetInProject(f.Table.Project, f.Table.Dataset)
	if _, err := dataset.Metadata(ctx); err != nil {
		return err
	}

	schema := mustInferSchema(f.Type.T)
	table := dataset.Table(f.Table.Table)
	if _, err := table.Metadata(ctx); err != nil {
		if !isNotFound(err) {
			return err
		}
		if f.Options.CreateDisposition == bigquery.CreateNever {
			return fmt.Errorf("table does not exist and create disposition is 'CreateNever': %v", err)
		}
		if err := table.Create(ctx, &bigquery.TableMetadata{Schema: schema}); err != nil {
			return err
		}
	}

	var data []reflect.Value
	// This stores the running byte size estimate of a BQ request.
	size := writeOverheadBytes

	var val beam.X
	for iter(&val) {
		current, err := getInsertSize(val.(any), schema)
		if err != nil {
			return errors.Wrapf(err, "bigquery write error")
		}
		if len(data)+1 > writeRowLimit || size+current > writeSizeLimit {
			// Write rows in batches to comply with BQ limits.

View on GitHub (pinned to 12126d8942)

Solutions

Set CreateDisposition to bigquery.CREATE_IF_NEEDED so Beam creates the table from the schema.
Create the table manually in BigQuery with the expected schema before running the pipeline.
Verify the table/dataset/project names are correct and that credentials can see the table.

Example fix

// before
bq.Write(s, project, "dataset.table", elems)
// after
opts := bq.WriteOptions{CreateDisposition: bigquery.CreateIfNeeded}
bq.WriteWithSchema(s, project, "dataset.table", schema, elems, opts)

Defensive patterns

Strategy: validation

Validate before calling

client, _ := bigquery.NewClient(ctx, projectID)
table := client.Dataset(dataset).Table(tableID)
if _, err := table.Metadata(ctx); err != nil && createDisposition == bigquery.CreateNever {
    return fmt.Errorf("destination table %s.%s missing and CreateNever set", dataset, tableID)
}

Try / catch

if err := beamJob.Run(ctx); err != nil && strings.Contains(err.Error(), "CreateNever") {
    // create table or switch disposition and retry
}

Prevention

Verify destination tables exist (or use CREATE_IF_NEEDED) before launching jobs.
Validate table names against the target project at pipeline-construction time.

When it happens

Trigger: Running bigqueryio.Write (or WriteWithSchema) with Options.CreateDisposition == bigquery.CreateNever while the destination table does not exist in the dataset.

Common situations: Typo in table name or dataset; writing to a different project than intended; table deleted before the job ran; forgetting the default CreateDisposition (CREATE_IF_NEEDED).

Understand the failure class

Background: 'Could not be found', 'does not exist', 'not found in database': the resource-not-found family when an ID, slug, key, or URI lookup comes back empty — this error's family across 20 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/a10566944f5eac76. Report an issue: GitHub.