apache/beam · error · ValueError

Duplicate column names found

Error message

Duplicate column names found: {duplicates}

What it means

PostgresVectorWriterConfig.__init__ validates that no two ColumnSpec entries share a column_name before generating the NamedTuple record type and SQL insert. Duplicate names would make the generated record type and INSERT ambiguous, so it fails fast.

Solutions

1. Give each ColumnSpec a unique column_name matching distinct Postgres columns.
2. De-duplicate the specs list before constructing the writer config.
3. Split genuinely different data into separate table columns rather than reusing a name.

Example fix

// before
config.add_vector("embedding").add_vector("embedding")
// after
config.add_vector("embedding").add_sparse_vector("sparse_embedding")

names = [c.column_name for c in specs]
assert len(names) == len(set(names)), f"duplicates: {set(n for n in names if names.count(n) > 1)}"

def has_unique_columns(specs) -> bool:
    names = [s.column_name for s in specs]
    return len(names) == len(set(names))

try:
    cfg = PostgresVectorWriterConfig(..., column_specs=specs)
except ValueError as e:
    if "Duplicate column names" in str(e):
        seen = set(); specs = [s for s in specs if not (s.column_name in seen or seen.add(s.column_name))]
        cfg = PostgresVectorWriterConfig(..., column_specs=specs)
    else:
        raise

When it happens

Trigger: Passing column_specs with repeated column_name values to PostgresVectorWriterConfig — e.g. two add_vector/add_metadata calls with the same column, or merged spec lists containing the same column twice.

Common situations: Programmatic spec generation loops reusing a name; both a dense and sparse spec accidentally named 'embedding'; config merging that concatenates specs without de-duplication.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• Duplicate column names found
• Expected chunk to contain embedding.
• Expected chunk to contain sparse embedding.
• Approximate Nearest Neighbor Search (ANNS) field must be…
• chunk_to_dict_fn is deprecated, use embeddable_to_dict_fn

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c9f60de9234bce83. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/postgres.py:58

class _PostgresQueryBuilder:
  def __init__(
      self,
      table_name: str,
      *,
      column_specs: list[ColumnSpec],
      conflict_resolution: Optional[ConflictResolution] = None):
    """Builds SQL queries for writing EmbeddableItems to Postgres.
    """
    self.table_name = table_name

    self.column_specs = column_specs
    self.conflict_resolution = conflict_resolution

    # Validate no duplicate column names
    names = [col.column_name for col in self.column_specs]
    duplicates = set(name for name in names if names.count(name) > 1)
    if duplicates:
      raise ValueError(f"Duplicate column names found: {duplicates}")

    # Create NamedTuple type
    fields = [(col.column_name, col.python_type) for col in self.column_specs]
    type_name = f"VectorRecord_{table_name}"
    self.record_type = NamedTuple(type_name, fields)  # type: ignore

    # Register coder
    registry.register_coder(self.record_type, RowCoder)

    # Set default update fields to all non-conflict fields if update fields are
    # not specified
    if self.conflict_resolution:
      self.conflict_resolution.maybe_set_default_update_fields(
          [col.column_name for col in self.column_specs if col.column_name])

  def build_insert(self) -> str:
    """Build INSERT query with proper type casting."""
    # Get column names and placeholders

View on GitHub (pinned to 12126d8942)