apache/beam · error · ValueError

Duplicate column names found

Error message

Duplicate column names found: {duplicates}

What it means

MySqlVectorWriterConfig.__init__ builds a NamedTuple record type from the ColumnSpec list, so each column name must be unique. If two specs share a column_name, the generated record type and SQL insert would be ambiguous, so it fails fast.

Solutions

1. Rename one of the duplicated column_name values in your ColumnSpec list.
2. Print [c.column_name for c in column_specs] and de-duplicate before constructing the writer config.
3. If two columns need different values but share a DB name, use distinct spec names or adjust the table schema.

Example fix

// before
specs = [ColumnSpec(id), ColumnSpec.vector("embedding"), ColumnSpec(id)]
// after
specs = [ColumnSpec(id), ColumnSpec.vector("embedding")]

names = [c.column_name for c in column_specs]
assert len(names) == len(set(names)), f"duplicate columns: {set(n for n in names if names.count(n) > 1)}"

def columns_unique(specs) -> bool:
    names = [c.column_name for c in specs]
    return len(names) == len(set(names))

try:
    cfg = MySqlVectorWriterConfig(..., column_specs=specs)
except ValueError as e:
    if "Duplicate column names" in str(e):
        seen = set(); specs = [s for s in specs if not (s.column_name in seen or seen.add(s.column_name))]
        cfg = MySqlVectorWriterConfig(..., column_specs=specs)
    else:
        raise

When it happens

Trigger: Passing column_specs where two ColumnSpec entries have the same column_name, e.g. a builder adding both a default id column and an explicit id column in MySqlVectorWriterConfig.

Common situations: Appending specs in a loop that reuses a fixed column name; combining metadata and vector fields that both map to 'content' or 'id'; merging two spec lists without de-duplicating.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Duplicate column names found
• Expected chunk to contain embedding.
• primary_key_field is required when action='IGNORE'
• Unknown conflict resolution
• Approximate Nearest Neighbor Search (ANNS) field must be…

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/48a66352283184ad. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/mysql.py:109

class _MySQLQueryBuilder:
  def __init__(
      self,
      table_name: str,
      *,
      column_specs: list[ColumnSpec],
      conflict_resolution: Optional[ConflictResolution] = None):
    """Builds SQL queries for writing EmbeddableItems with Embeddings to MySQL.
    """
    self.table_name = table_name

    self.column_specs = column_specs
    self.conflict_resolution_strategy = _create_conflict_strategy(
        conflict_resolution)

    names = [col.column_name for col in self.column_specs]
    duplicates = set(name for name in names if names.count(name) > 1)
    if duplicates:
      raise ValueError(f"Duplicate column names found: {duplicates}")

    fields = [(col.column_name, col.python_type) for col in self.column_specs]
    type_name = f"VectorRecord_{table_name}"
    self.record_type = NamedTuple(type_name, fields)  # type: ignore

    registry.register_coder(self.record_type, RowCoder)

  def build_insert(self) -> str:
    fields = [col.column_name for col in self.column_specs]
    placeholders = [col.placeholder for col in self.column_specs]

    # Build base query
    query = f"""
        INSERT INTO {self.table_name}
        ({', '.join(fields)})
        VALUES ({', '.join(placeholders)})
    """
    conflict_clause = self.conflict_resolution_strategy.get_conflict_clause(

View on GitHub (pinned to 12126d8942)