apache/beam · error · ValueError

Shard number pattern S+ not found in shard_name_template

Error message

Shard number pattern S+ not found in shard_name_template: %s

What it means

FileBasedSink._template_replace_shard_num rewrites the S+ pattern in a shard name template into a numeric format specifier; if the template contains no 'S' sequence, a ValueError is raised because a shard number placeholder is mandatory.

Solutions

1. Include an S pattern in the template, e.g. '-SSSS-of-NNNN'
2. Use the default shard template (DEFAULT_SHARD_NAME_TEMPLATE)
3. Pass shard_name_template=None to get default behavior instead of an S-less string

Example fix

// before
WriteToText('/out/data', shard_name_template='-of-NN')
// after
WriteToText('/out/data', shard_name_template='-SSS-of-NN')

import re
if shard_name_template and not re.search('S+', shard_name_template):
    raise ValueError(f'shard_name_template must contain an S+ pattern: {shard_name_template!r}')

def has_shard_pattern(t: str) -> bool:
    return re.search(r'S+', t) is not None

When it happens

Trigger: Passing a shard_name_template such as '-of-N' or 'part-' without any 'S' characters (e.g. shard_name_template='') to a sink or to _template_replace_shard_num directly.

Common situations: Custom shard templates edited to remove the S token; templates copied from systems using different placeholders.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• A schema is required to write non-schema'd data.
• A sink must inherit iobase.Sink, iobase.NativeSink, or be a…
• An explicit schema is required to write non-schema'd…
• Cannot call read after iterating.
• Cannot create a temporary directory for root path prefix

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/f05454d4d40b9e91. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/filebasedsink.py:498

    if match:
      shard_name_template = shard_name_template.replace(
          match.group(0), '%%(uuid)0%dd' % len(match.group(0)))
    return shard_name_template

  @staticmethod
  def _template_replace_num_shards(shard_name_template):
    match = re.search('N+', shard_name_template)
    if match:
      shard_name_template = shard_name_template.replace(
          match.group(0), '%%(num_shards)0%dd' % len(match.group(0)))
    return shard_name_template

  @staticmethod
  def _template_replace_shard_num(shard_name_template):
    match = re.search('S+', shard_name_template)
    if match is None:
      # shard name is required in the template.
      raise ValueError(
          "Shard number pattern S+ not found in shard_name_template: %s" %
          shard_name_template)
    return shard_name_template.replace(
        match.group(0), '%%(shard_num)0%dd' % len(match.group(0)))

  @staticmethod
  def _template_to_format(shard_name_template):
    if not shard_name_template:
      return ''
    # shard_num is required in the template, while others are optional.
    replace_funcs = [
        FileBasedSink._template_replace_shard_num,
        FileBasedSink._template_replace_num_shards,
        FileBasedSink._template_replace_uuid,
        FileBasedSink._template_replace_window
    ]
    for func in replace_funcs:
      shard_name_template = func(shard_name_template)

View on GitHub (pinned to 12126d8942)