pathwaycom/pathway · error · ValueError

'json_field_paths' references field {field_name!r} which is

Error message

'json_field_paths' references field {field_name!r} which is not in the schema. Known fields: {sorted(schema_columns)}.

What it means

Raised by Pathway's input-connector helpers when the 'json_field_paths' dict maps a field name that is not a column of the parsed schema. 'json_field_paths' tells the connector where each output column's value lives inside the incoming JSON document, so every key of that dict must match a column name produced by the schema (the connector's schema or the one you supplied). The error lists all known field names to make the mismatch obvious.

Source

Thrown at python/pathway/io/_utils.py:387

            raise ValueError("Unexpected argument for json format: csv_settings")
        if autogenerate_key:
            raise ValueError(
                f"'autogenerate_key' is only meaningful for 'raw' or "
                f"'plaintext' formats and would have no effect with "
                f"{format!r}. Drop it or pick a compatible format."
            )
        if json_field_paths is not None:
            schema_columns = set(schema.column_names())
            for field_name, path in json_field_paths.items():
                if field_name == METADATA_COLUMN_NAME:
                    raise ValueError(
                        f"'json_field_paths' cannot be used for "
                        f"{METADATA_COLUMN_NAME!r}: the connector populates "
                        f"this column itself when 'with_metadata=True', so "
                        f"any JSON path would be silently ignored."
                    )
                if field_name not in schema_columns:
                    raise ValueError(
                        f"'json_field_paths' references field {field_name!r} "
                        f"which is not in the schema. Known fields: "
                        f"{sorted(schema_columns)}."
                    )
                if path != "" and not path.startswith("/"):
                    raise ValueError(
                        f"Invalid JSON Pointer for field {field_name!r}: "
                        f"{path!r}. JSON Pointers (RFC 6901) must be empty "
                        f"or start with '/' (e.g. '/foo/bar')."
                    )
        return schema, api.DataFormat(
            **api_schema,
            format_type=data_format_type,
            column_paths=json_field_paths,
            schema_registry_settings=maybe_schema_registry_settings(
                schema_registry_settings
            ),
        )

View on GitHub (pinned to fa2f74a464)

Solutions

1. Compare the keys of json_field_paths against the sorted list of known fields printed in the error message and fix or remove the offending entry.
2. If you intended the field to exist, add it to the schema passed to the connector (or adjust the connector's schema definition) so the name matches.
3. If you meant to extract a nested value, keep the field name identical to the schema column and only change the JSON Pointer path value.

Example fix

# before
schema = tb.schema_from_types(user=str, amount=int)
t = pw.io.kafka.read(..., schema=schema,
    json_field_paths={'username': '/user'})  # 'username' not in schema

# after
schema = tb.schema_from_types(username=str, amount=int)
t = pw.io.kafka.read(..., schema=schema,
    json_field_paths={'username': '/user'})  # key matches schema column

def validate_json_field_paths(schema, json_field_paths):
    known = set(schema.column_names())
    unknown = set(json_field_paths) - known
    assert not unknown, f"json_field_paths fields not in schema: {sorted(unknown)}; known: {sorted(known)}"

def paths_match_schema(schema, json_field_paths: dict) -> bool:
    return set(json_field_paths).issubset(set(schema.column_names()))

try:
    t = pw.io.kafka.read(..., schema=schema, json_field_paths=paths)
except ValueError as e:
    if "not in the schema" in str(e):
        raise SystemExit(f"Fix json_field_paths keys: {e}")
    raise

When it happens

Trigger: Calling an input connector (e.g. pw.io.kafka.read, pw.io.http.read) with both a schema and json_field_paths={'payload_user': '/user'} where 'payload_user' is not a column of that schema; or renaming/removing a schema column without updating json_field_paths; or misspelling a field name in the mapping.

Common situations: Copy-pasting a json_field_paths dict from an example whose schema differs from yours; evolving the schema (column renamed) while leaving stale entries in json_field_paths; assuming json_field_paths creates new columns instead of only binding existing ones.

Related errors

• parameters `schema` and `id_from` are mutually exclusive
• Failed to detect the region of S3 bucket {bucket!r} (HTTP st
• Schema.with_types() argument name has to be an existing colu
• Schema.without() argument {name!r} has to refer to an existi
• schema does not have columns {missing_columns}

AI-assisted analysis of pathwaycom/pathway@fa2f74a464 (2026-08-15). Data as JSON: /api/errors/ee593bce3c81f03b. Report an issue: GitHub.