{"record":{"id":"c95738c30751388c","repo":"apache/beam","slug":"primary-key-validation-error-multiple-inserts-with-the-same","errorCode":null,"errorMessage":"Primary key validation error! Multiple inserts with the same primary key.","messagePattern":"Primary key validation error! Multiple inserts with the same primary key\\.","errorType":"validation","errorClass":"RuntimeException","httpStatus":null,"severity":"error","filePath":"sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java","lineNumber":115,"sourceCode":"      List<Object> cellValues =\n          ((List<AbstractMap<String, Object>>) fValue)\n              .stream()\n                  .map(cell -> Preconditions.checkStateNotNull(cell.get(\"v\")))\n                  .collect(Collectors.toList());\n\n      return Preconditions.checkStateNotNull(primaryKeyColumnIndices).stream()\n          .map(cellValues::get)\n          .collect(Collectors.toList());\n    } else {\n      return primaryKeyColumns.stream().map(tableRow::get).collect(Collectors.toList());\n    }\n  }\n\n  long addRow(TableRow row, String id) {\n    List<Object> primaryKey = getPrimaryKey(row);\n    if (primaryKey != null && !primaryKey.isEmpty()) {\n      if (keyedRows.putIfAbsent(primaryKey, row) != null) {\n        throw new RuntimeException(\n            \"Primary key validation error! Multiple inserts with the same primary key.\");\n      }\n    } else {\n      rows.add(row);\n      if (id != null) {\n        ids.add(id);\n      }\n    }\n\n    long tableSize = table.getNumBytes() == null ? 0L : table.getNumBytes();\n    try {\n      long rowSize = TableRowJsonCoder.of().getEncodedElementByteSize(row);\n      table.setNumBytes(tableSize + rowSize);\n      return rowSize;\n    } catch (Exception ex) {\n      throw new RuntimeException(\"Failed to convert the row to JSON\", ex);\n    }\n  }","sourceCodeStart":97,"sourceCodeEnd":133,"githubUrl":"https://github.com/apache/beam/blob/12126d8942aaf848030c478b4c6a28c6af861c66/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java#L97-L133","documentation":"FakeBigQueryServices' TableContainer enforces primary-key uniqueness for tables that define a primary key. addRow throws this RuntimeException when putIfAbsent detects a row already exists for the same key. It mimics BigQuery primary-key (DML insert) semantics in the testing fake.","triggerScenarios":"Calling insertAll with two rows having identical primary-key values when the fake table schema declares a primary key.","commonSituations":"Pipeline tests that produce duplicate records (unkeyed grouping, replayed input, non-idempotent writes) then fail only in tests using the fake BigQuery service.","solutions":["Deduplicate records before insertAll (e.g. Deduplicate or GBK by primary key)","Make inserts idempotent (upsert via MERGE-like DML instead of INSERT)","If duplicates are expected, test against a fake table without a primary key","Log the offending primary key from the failing row to find the duplicate source"],"exampleFix":"// before\ntableRowWriter.write(record); // duplicate PKs reach insertAll\n// after\nPipeline p = ...;\nPCollection<TableRow> deduped = rows.apply(\"dedup\",\n    Deduplicate.<TableRow>keyedBy(t -> getPrimaryKey(t)));","handlingStrategy":"validation","validationCode":"Set<List<Object>> seen = new HashSet<>();\nfor (TableRow row : batch) {\n  if (!seen.add(getPrimaryKey(row))) {\n    throw new IllegalArgumentException(\"duplicate PK in batch: \" + getPrimaryKey(row));\n  }\n}","typeGuard":null,"tryCatchPattern":"try {\n  container.insertAll(rows);\n} catch (RuntimeException e) {\n  if (e.getMessage().contains(\"same primary key\")) {\n    // deduplicate and retry\n  } else throw e;\n}","preventionTips":["Deduplicate input before insertAll in tests","Key pipelines on the primary key before writing","Test against fake tables without PKs when duplicates are expected"],"tags":["java","bigquery","testing","duplicate-key","validation"],"backgroundTag":"schema-validation-failed","analyzedSha":"12126d8942aaf848030c478b4c6a28c6af861c66","analyzedAt":"2026-09-13T01:50:10.254Z","contentChangedAt":"2026-09-13T01:50:10.254Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}