apache/beam · error · ValueError

should be either a single PCollection or a dict of named…

Error message

{} should be either a single PCollection or a dict of named PCollections.

What it means

SqlChain.expand accepts either a single PCollection or a dict of named PCollections as the SQL source; any other type (list, None, string, etc.) raises ValueError telling the caller the source shape is invalid.

Solutions

1. Wrap multiple PCollections in a dict: {'name1': pcoll1, 'name2': pcoll2}
2. Pass a single PCollection directly if the query uses one input
3. Check that the variable you pass is a beam.PCollection (print its type)
4. Resolve PCollections by name via pcoll_by_name() and confirm each lookup succeeded

Example fix

// before
chain = SqlChain(query, source=[pcoll1, pcoll2])
// after
chain = SqlChain(query, source={'pcoll1': pcoll1, 'pcoll2': pcoll2})

def valid_sql_source(source):
    import beam as _b
    return isinstance(source, _b.pvalue.PCollection) or (
        isinstance(source, dict) and all(isinstance(v, _b.pvalue.PCollection) for v in source.values()))

def is_sql_source(source):
    return isinstance(source, beam.pvalue.PCollection) or (
        isinstance(source, dict) and source and
        all(isinstance(v, beam.pvalue.PCollection) for v in source.values()))

try:
    chain.to_pipeline()
except ValueError as e:
    if 'should be either a single PCollection' in str(e):
        source = {'main': source_or_lookup}

When it happens

Trigger: Constructing a SqlChain (or calling find_sql_source_based_inputs / expand) with a source that is neither beam.PCollection nor dict, e.g. passing the result of a lookup that returned None or a list of PCollections.

Common situations: Programmatic use of SqlChain outside the %%beam_sql magic; passing multiple PCollections as a list instead of a dict keyed by name; a variable holding a non-PCollection after a failed lookup.

Understand the failure class

Background: Type mismatch errors: IllegalArgumentException, TypeError and type guards across 150 open-source libraries — this error's family across 150 libraries.

Related errors

• Attempting to alter catalog
• Attempting to create catalog
• Attempting to create database
• Attempting to drop a table using unexpected Calcite Schema…
• failed to scan

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/602f3f03ae091c28. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/runners/interactive/sql/sql_chain.py:174

  def expand(self, source):
    """Applies the SQL transform. If a PCollection uses a schema defined in
    the main session, use the additional DoFn to restore it on the worker."""
    if isinstance(source, dict):
      schema_loaded = {
          tag: pcoll | 'load_schemas_{}_tag_{}_{}'.format(
              self.output_name, tag, self.execution_count) >> beam.ParDo(
                  self._SqlTransformDoFn(self.schemas, self.schema_annotations))
          if pcoll.element_type in self.schemas else pcoll
          for tag, pcoll in source.items()
      }
    elif isinstance(source, beam.pvalue.PCollection):
      schema_loaded = source | 'load_schemas_{}_{}'.format(
          self.output_name, self.execution_count) >> beam.ParDo(
              self._SqlTransformDoFn(self.schemas, self.schema_annotations)
          ) if source.element_type in self.schemas else source
    else:
      raise ValueError(
          '{} should be either a single PCollection or a dict of named '
          'PCollections.'.format(source))
    return schema_loaded | 'beam_sql_{}_{}'.format(
        self.output_name, self.execution_count) >> SqlTransform(self.query)


@dataclass
class SqlChain:
  """A chain of SqlNodes.

  Attributes:
    nodes: all nodes by their output_names.
    root: the first SqlNode applied chronologically.
    current: the last node applied.
    user_pipeline: the user defined pipeline this chain originates from. If
      None, the whole chain just computes from raw values in queries.
      Otherwise, at least some of the nodes in chain has queried against
      PCollections.

View on GitHub (pinned to 12126d8942)