apache/beam · error · ValueError

Duplicate column names

Error message

Duplicate column names: {duplicates}

What it means

SpannerVectorRecord (the writer's table mapping) is built from column_specs, and its NamedTuple record type cannot have two fields with the same name. __init__ scans spec names for duplicates and raises at construction time with the duplicated set.

Solutions

1. Remove or rename the duplicate spec so each column_name appears once
2. Dedupe specs before constructing: {spec.column_name: spec for spec in specs}.values()
3. If two embeddings are needed, use distinct column names (e.g. 'embedding_dense', 'embedding_sparse')

Example fix

// before
builder.with_embedding_spec('embedding').with_embedding_spec('embedding', convert_fn=f)
// after
builder.with_embedding_spec('embedding').with_embedding_spec('embedding_v2', convert_fn=f)

names = [c.column_name for c in column_specs]
dupes = {n for n in names if names.count(n) > 1}
assert not dupes, f"Duplicate Spanner columns: {dupes}"

def specs_are_unique(specs) -> bool:
    names = [c.column_name for c in specs]
    return len(names) == len(set(names))

When it happens

Trigger: Building SpannerVectorRecord with a list of SpannerColumnSpec where two specs share column_name — e.g. adding two embedding specs with the same column name, or programmatically merging spec lists that overlap.

Common situations: Calling with_embedding_spec twice for the same column; combining default id/content specs with custom specs that repeat a name; joining column lists from multiple sources without deduping.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

• A schema is required to write non-schema'd data.
• All dicts in batch must have the same keys. extra keys
• An explicit schema is required to write non-schema'd…
• Arrow map key field cannot be nullable
• Attempted to encode null for non-nullable field

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/9b431db25acf00d2. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/ml/rag/ingestion/spanner.py:435

  """
  def __init__(self, table_name: str, column_specs: list[SpannerColumnSpec]):
    """Initialize schema builder.
    
    Args:
        table_name: Table name (used in NamedTuple type name)
        column_specs: List of column specifications
    
    Raises:
        ValueError: If duplicate column names are found
    """
    self.table_name = table_name
    self.column_specs = column_specs

    # Validate no duplicates
    names = [col.column_name for col in column_specs]
    duplicates = set(name for name in names if names.count(name) > 1)
    if duplicates:
      raise ValueError(f"Duplicate column names: {duplicates}")

    # Create NamedTuple type
    fields = [(col.column_name, col.python_type) for col in column_specs]
    type_name = f"SpannerVectorRecord_{table_name}"
    self.record_type = NamedTuple(type_name, fields)  # type: ignore

    # Register coder
    registry.register_coder(self.record_type, RowCoder)

  def create_converter(self) -> Callable[[EmbeddableItem], NamedTuple]:
    """Create converter function from EmbeddableItem to NamedTuple record.

    Returns:
        Function that converts an EmbeddableItem to a NamedTuple record
    """
    def convert(embeddable: EmbeddableItem) -> self.record_type:  # type: ignore
      values = {
          col.column_name: col.value_fn(embeddable)

View on GitHub (pinned to 12126d8942)