apache/beam · error · ValueError

Encountered multiple indexes with the name '%s'. Cannot conv

Error message

Encountered multiple indexes with the name '%s'. Cannot convert to a schema-aware PCollection with include_indexes=True. Please ensure all indexes have unique names or consider not including indexes.

What it means

With include_indexes=True, index levels become schema fields, so their names must be unique. If two index levels share the same name, the schema would contain duplicate field names, and ValueError is raised listing the duplicated name.

Source

Thrown at sdks/python/apache_beam/dataframe/schemas.py:149


def element_typehint_from_dataframe_proxy(
    proxy: pd.DataFrame, include_indexes: bool = False) -> RowTypeConstraint:

  output_columns = []
  if include_indexes:
    remaining_index_names = list(proxy.index.names)
    i = 0
    while len(remaining_index_names):
      index_name = remaining_index_names.pop(0)
      if index_name is None:
        raise ValueError(
            "Encountered an unnamed index. Cannot convert to a "
            "schema-aware PCollection with include_indexes=True. "
            "Please name all indexes or consider not including "
            "indexes.")
      elif index_name in remaining_index_names:
        raise ValueError(
            "Encountered multiple indexes with the name '%s'. "
            "Cannot convert to a schema-aware PCollection with "
            "include_indexes=True. Please ensure all indexes have "
            "unique names or consider not including indexes." % index_name)
      elif index_name in proxy.columns:
        raise ValueError(
            "Encountered an index that has the same name as one "
            "of the columns, '%s'. Cannot convert to a "
            "schema-aware PCollection with include_indexes=True. "
            "Please ensure all indexes have unique names or "
            "consider not including indexes." % index_name)
      else:
        # its ok!
        output_columns.append(
            (index_name, proxy.index.get_level_values(i).dtype))
        i += 1

  output_columns.extend(zip(proxy.columns, proxy.dtypes))

View on GitHub (pinned to 12126d8942)

Solutions

  1. Rename the duplicate index levels to unique names (df.index.names = ['l1','l2', ...])
  2. Drop redundant levels via df.droplevel(i) if they duplicate information
  3. Use reset_index() to flatten the index and disambiguate column names, then convert without include_indexes

Example fix

// before
df.index.names = ['id', 'id']
// after
df.index.names = ['id_level_1', 'id_level_2']
Defensive patterns

Strategy: validation

Validate before calling

names = [n for n in proxy.index.names if n is not None]
if len(names) != len(set(names)):
    dupes = {n for n in names if names.count(n) > 1}
    raise ValueError(f'Duplicate index names: {dupes}; rename before include_indexes=True')

Type guard

def has_unique_index_names(df) -> bool:
    names = [n for n in df.index.names if n is not None]
    return len(names) == len(set(names))

Try / catch

try:
    pc = beam.dataframe.convert.to_pcollection(df, include_indexes=True)
except ValueError as e:
    if 'multiple indexes with the name' in str(e):
        df.index.names = [f'index_{i}_{n}' for i, n in enumerate(df.index.names)]
        pc = beam.dataframe.convert.to_pcollection(df, include_indexes=True)

Prevention

When it happens

Trigger: Converting a DataFrame with a MultiIndex where two levels have identical names, using include_indexes=True via element_typehint_from_dataframe_proxy / to_pcollection.

Common situations: MultiIndexes built from reset/index operations where levels kept the same source column name (e.g. groupby on the same column twice).

Understand the failure class

Background: Schema validation failed / invalid input schema: payload rejected because its shape doesn't match the expected schema — this error's family across 28 libraries.

Related errors

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/664d6cb0d581c78e. Report an issue: GitHub.