apache/beam · error · ValueError

Encountered an index that has the same name as one of the co

Error message

Encountered an index that has the same name as one of the columns, '%s'. Cannot convert to a schema-aware PCollection with include_indexes=True. Please ensure all indexes have unique names or consider not including indexes.

What it means

With include_indexes=True, both index levels and columns become schema fields, so an index name equal to a column name would produce a duplicate field. ValueError is raised naming the conflicting identifier.

Source

Thrown at sdks/python/apache_beam/dataframe/schemas.py:155

  if include_indexes:
    remaining_index_names = list(proxy.index.names)
    i = 0
    while len(remaining_index_names):
      index_name = remaining_index_names.pop(0)
      if index_name is None:
        raise ValueError(
            "Encountered an unnamed index. Cannot convert to a "
            "schema-aware PCollection with include_indexes=True. "
            "Please name all indexes or consider not including "
            "indexes.")
      elif index_name in remaining_index_names:
        raise ValueError(
            "Encountered multiple indexes with the name '%s'. "
            "Cannot convert to a schema-aware PCollection with "
            "include_indexes=True. Please ensure all indexes have "
            "unique names or consider not including indexes." % index_name)
      elif index_name in proxy.columns:
        raise ValueError(
            "Encountered an index that has the same name as one "
            "of the columns, '%s'. Cannot convert to a "
            "schema-aware PCollection with include_indexes=True. "
            "Please ensure all indexes have unique names or "
            "consider not including indexes." % index_name)
      else:
        # its ok!
        output_columns.append(
            (index_name, proxy.index.get_level_values(i).dtype))
        i += 1

  output_columns.extend(zip(proxy.columns, proxy.dtypes))

  fields = [(column, dtype_to_fieldtype(dtype))
            for (column, dtype) in output_columns]
  field_options: Optional[dict[str, Sequence[tuple[str, Any]]]]
  if include_indexes:
    field_options = {

View on GitHub (pinned to 12126d8942)

Solutions

Rename the index (df.index.name = 'index_id') so it differs from all column names
Drop the index from inclusion with include_indexes=False, or reset_index(drop=True)
Remove/rename the conflicting column before conversion

Example fix

// before
df.index.name = 'id'; df['id'] = ...
// after
df.index.name = 'row_index'  # distinct from column 'id'

Defensive patterns

Strategy: validation

Validate before calling

if include_indexes and proxy.index.name is not None and proxy.index.name in proxy.columns:
    proxy.index.name = proxy.index.name + '_index'  # avoid collision with column field

Type guard

def index_names_disjoint_from_columns(df) -> bool:
    return all(n is None or n not in df.columns for n in df.index.names)

Try / catch

try:
    pc = beam.dataframe.convert.to_pcollection(df, include_indexes=True)
except ValueError as e:
    if 'same name as one of the columns' in str(e):
        df.index.name = f'{df.index.name}_index'
        pc = beam.dataframe.convert.to_pcollection(df, include_indexes=True)

Prevention

Keep index names distinct from column names by convention (e.g. suffix '_index')
Run a name-collision check before any include_indexes=True conversion
Consider reset_index() and unique column names as the canonical schema

When it happens

Trigger: Converting to a schema-aware PCollection with include_indexes=True where df.index.name equals one of df.columns (e.g. index named 'id' and a column 'id').

Common situations: reset_index-style workflows where the index kept the column's name; joins/groupbys on a key column that also became the index.

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/1021cb4d6f6fda46. Report an issue: GitHub.