apache/beam · error · RuntimeException

Primary key validation error! Multiple inserts with the same

Error message

Primary key validation error! Multiple inserts with the same primary key.

What it means

FakeBigQueryServices' TableContainer enforces primary-key uniqueness for tables that define a primary key. addRow throws this RuntimeException when putIfAbsent detects a row already exists for the same key. It mimics BigQuery primary-key (DML insert) semantics in the testing fake.

Source

Thrown at sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/TableContainer.java:115

      List<Object> cellValues =
          ((List<AbstractMap<String, Object>>) fValue)
              .stream()
                  .map(cell -> Preconditions.checkStateNotNull(cell.get("v")))
                  .collect(Collectors.toList());

      return Preconditions.checkStateNotNull(primaryKeyColumnIndices).stream()
          .map(cellValues::get)
          .collect(Collectors.toList());
    } else {
      return primaryKeyColumns.stream().map(tableRow::get).collect(Collectors.toList());
    }
  }

  long addRow(TableRow row, String id) {
    List<Object> primaryKey = getPrimaryKey(row);
    if (primaryKey != null && !primaryKey.isEmpty()) {
      if (keyedRows.putIfAbsent(primaryKey, row) != null) {
        throw new RuntimeException(
            "Primary key validation error! Multiple inserts with the same primary key.");
      }
    } else {
      rows.add(row);
      if (id != null) {
        ids.add(id);
      }
    }

    long tableSize = table.getNumBytes() == null ? 0L : table.getNumBytes();
    try {
      long rowSize = TableRowJsonCoder.of().getEncodedElementByteSize(row);
      table.setNumBytes(tableSize + rowSize);
      return rowSize;
    } catch (Exception ex) {
      throw new RuntimeException("Failed to convert the row to JSON", ex);
    }
  }

View on GitHub (pinned to 12126d8942)

Solutions

Deduplicate records before insertAll (e.g. Deduplicate or GBK by primary key)
Make inserts idempotent (upsert via MERGE-like DML instead of INSERT)
If duplicates are expected, test against a fake table without a primary key
Log the offending primary key from the failing row to find the duplicate source

Example fix

// before
tableRowWriter.write(record); // duplicate PKs reach insertAll
// after
Pipeline p = ...;
PCollection<TableRow> deduped = rows.apply("dedup",
    Deduplicate.<TableRow>keyedBy(t -> getPrimaryKey(t)));

Defensive patterns

Strategy: validation

Validate before calling

Set<List<Object>> seen = new HashSet<>();
for (TableRow row : batch) {
  if (!seen.add(getPrimaryKey(row))) {
    throw new IllegalArgumentException("duplicate PK in batch: " + getPrimaryKey(row));
  }
}

Try / catch

try {
  container.insertAll(rows);
} catch (RuntimeException e) {
  if (e.getMessage().contains("same primary key")) {
    // deduplicate and retry
  } else throw e;
}

Prevention

Deduplicate input before insertAll in tests
Key pipelines on the primary key before writing
Test against fake tables without PKs when duplicates are expected

When it happens

Trigger: Calling insertAll with two rows having identical primary-key values when the fake table schema declares a primary key.

Common situations: Pipeline tests that produce duplicate records (unkeyed grouping, replayed input, non-idempotent writes) then fail only in tests using the fake BigQuery service.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c95738c30751388c. Report an issue: GitHub.