apache/beam · error · ValueError

Unable to parse JSON schema

Error message

Unable to parse JSON schema: %s - %r

What it means

Raised by parse_table_schema_from_json when the schema string passed in is not valid JSON (json.loads raises JSONDecodeError). Beam wraps it in a ValueError including the original string and the decoder error so the malformed schema can be located.

Solutions

1. Validate the string with json.loads(schema_string) locally to see the exact JSONDecodeError position.
2. Convert Python dict literals to valid JSON using double quotes: json.dumps(python_dict) before storing/passing.
3. If you have a dict already, pass it via the dict-based schema path or call json.dumps on it first.
4. Check the source (file/env var/CLI arg) isn't truncated or wrapped in quotes.

Example fix

# before
parse_table_schema_from_json("{'fields': [{'name': 'x', 'type': 'STRING'}]}")

# after
parse_table_schema_from_json('{"fields": [{"name": "x", "type": "STRING"}]}')

import json
def validate_schema_string(s):
    if isinstance(s, str):
        json.loads(s)  # raises before Beam does

def is_valid_json(s):
    if not isinstance(s, str):
        return False
    try:
        json.loads(s)
        return True
    except ValueError:
        return False

try:
    schema = parse_table_schema_from_json(s)
except ValueError as e:
    if 'Unable to parse JSON schema' in str(e):
        raise ConfigError('schema string must be valid JSON (double quotes)') from e
    raise

When it happens

Trigger: Calling parse_table_schema_from_json(schema_string) with a non-JSON string — e.g. a Python-dict repr, single-quoted JSON, trailing commas, or a stringified dict from logging output.

Common situations: Pasting a schema dict from Python repr form (single quotes) into config; using WriteToBigQuery(schema='...') where the string is expected to be JSON but users pass a Python dict literal; reading a schema from an env var or file that was corrupted.

Understand the failure class

Background: JSON parse error: "Unexpected token" / "not valid JSON" / "failed to parse" — what JSON parsers are really complaining about — this error's family across 45 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Error while converting table schema
• bigquery write error
• Both a query and an output type of 'BEAM_ROW' were…
• Conflicting field modes for field
• Conflicting field types for field

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/bb82465e4d4573d0. Report an issue: GitHub.

Appendix: source

Thrown at sdks/python/apache_beam/io/gcp/bigquery_tools.py:216

  """
  table_ref = table_ref_elem_kv[0]
  hashable_table_ref = get_hashable_destination(table_ref)
  return (hashable_table_ref, table_ref_elem_kv[1])


def parse_table_schema_from_json(schema_string):
  """Parse the Table Schema provided as string.

  Args:
    schema_string: String serialized table schema, should be a valid JSON.

  Returns:
    A TableSchema of the BigQuery export from either the Query or the Table.
  """
  try:
    json_schema = json.loads(schema_string)
  except JSONDecodeError as e:
    raise ValueError(
        'Unable to parse JSON schema: %s - %r' % (schema_string, e))

  def _parse_schema_field(field):
    """Parse a single schema field from dictionary.

    Args:
      field: Dictionary object containing serialized schema.

    Returns:
      A TableFieldSchema for a single column in BigQuery.
    """
    schema = bigquery.TableFieldSchema()
    schema.name = field['name']
    schema.type = field['type']
    if 'mode' in field:
      schema.mode = field['mode']
    else:
      schema.mode = 'NULLABLE'

View on GitHub (pinned to 12126d8942)