{"record":{"id":"8eda51a19c4130f5","repo":"apache/beam","slug":"failed-to-patch-table-schema","errorCode":null,"errorMessage":"Failed to patch table schema.","messagePattern":"Failed to patch table schema\\.","errorType":"exception","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PatchTableSchemaDoFn.java","lineNumber":159,"sourceCode":"          o.output(KV.of(destination, null));\n          return;\n        } catch (IOException e) {\n          ApiErrorExtractor errorExtractor = new ApiErrorExtractor();\n          if (errorExtractor.preconditionNotMet(e) || errorExtractor.badRequest(e)) {\n            schemaOutOfDate = true;\n            break;\n          } else {\n            lastException = e;\n          }\n        }\n      } while (BackOffUtils.next(Sleeper.DEFAULT, backoff));\n      if (schemaOutOfDate) {\n        // This could be due to an out-of-date schema.\n        LOG.info(\"Schema out of date. Refreshing.\");\n        messageConverter.updateSchemaFromTable();\n      } else {\n        // We ran out of retries.\n        throw new RuntimeException(\"Failed to patch table schema.\", lastException);\n      }\n    }\n  }\n}\n","sourceCodeStart":141,"sourceCodeEnd":164,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/PatchTableSchemaDoFn.java#L141-L164","documentation":"This RuntimeException is thrown by PatchTableSchemaDoFn when the BigQuery Storage Write patch-schema retry loop is exhausted without success. The DoFn attempts to patch the table schema via the Storage API; if the failure is due to an out-of-date cached schema it refreshes the schema from the table and retries, but once all retries are consumed the last underlying exception is rethrown wrapped in this message.","triggerScenarios":"The table schema patch request fails repeatedly (network/gRPC errors, permission problems, or incompatible schema changes) until the configured BackOff retries are exhausted and schemaOutOfDate is false.","commonSituations":"Service account lacking bigquery.tables.update permission; schema change rejected as incompatible (e.g. removing required fields, changing field modes); transient gRPC outages lasting longer than the retry budget; concurrent writers patching the same table schema.","solutions":["Check the wrapped lastException cause to find the underlying gRPC/API error","Verify the service account has bigquery.tables.update permission on the dataset","Inspect the target table schema and confirm your Beam schema is a compatible patch (additive changes only, no field removals/type narrowing)","Increase retry backoff duration or re-run the pipeline after resolving transient issues","Resolve concurrent schema edits by other jobs writing to the same table"],"exampleFix":"// before: pipeline fails on patch retries exhausted\n// after: verify table is updatable and schema is additive before running\nTableSchema existing = bigquery.getTable(tableId).getSchema();\nList<String> existingFields = existing.getFieldsList().stream().map(TableFieldSchema::getName).collect(Collectors.toList());\nif (!newFieldNames.containsAll(existingFields)) {\n  throw new IllegalArgumentException(\"Patch must only add fields\");\n}","handlingStrategy":"retry","validationCode":"// Verify table schema is patchable before running the pipeline\nTable tbl = bigquery.getTable(datasetId, tableId);\nSet<String> existing = tbl.getDefinition().getSchema().getFields().stream().map(Field::getName).collect(Collectors.toSet());\nSet<String> planned = beamSchema.getFields().stream().map(Schema.Field::getName).collect(Collectors.toSet());\nif (!existing.containsAll(planned)) throw new IllegalStateException(\"Patch removes/renames fields: \" + existing.removeAll(planned));","typeGuard":"boolean isCompatiblePatch(TableSchema current, TableSchema patch) {\n  Set<String> names = current.getFieldsList().stream().map(TableFieldSchema::getName).collect(Collectors.toSet());\n  return patch.getFieldsList().stream().allMatch(f -> !names.contains(f.getName()) || compatible(f, names));\n}","tryCatchPattern":"try {\n  runPipeline();\n} catch (RuntimeException e) {\n  if (e.getMessage() != null && e.getMessage().contains(\"Failed to patch table schema\")) {\n    log.error(\"Schema patch retries exhausted; cause:\", e.getCause());\n    // inspect cause gRPC status, fix permissions/schema, then resubmit\n  } else { throw e; }\n}","preventionTips":["Grant bigquery.tables.update on the dataset to the pipeline service account","Restrict schema patches to additive changes","Avoid multiple jobs patching the same table schema concurrently","Monitor the wrapped cause's gRPC status in logs before re-running"],"tags":["bigquery","storage-write","schema","retry-exhausted","gcp"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}