What causes "Some field is missing in the JSON data." in apache/flink?

Set 'json.fail-on-missing-field'='false' (absent fields become null) if absence is legitimate

apache/flink · error · org.apache.flink.formats.json.JsonParseException

Some field is missing in the JSON data.

Error message

Some field is missing in the JSON data.

What it means

JsonParseException from createRowConverter after the object is fully parsed: fewer known fields were present than the row arity while 'json.fail-on-missing-field'='true'. It enforces the strict policy you opted into: every declared column must appear in the JSON object (extra unknown fields are skipped, missing ones are fatal).

Source

Thrown at flink-formats/flink-json/src/main/java/org/apache/flink/formats/json/JsonParserToRowDataConverters.java:436

                String fieldName = jp.getText();
                jp.nextToken();
                Integer idx = nameIdxMap.get(fieldName);
                if (idx != null) {
                    try {
                        Object convertField = fieldConverters[idx].convert(jp);
                        row.setField(idx, convertField);
                    } catch (Throwable t) {
                        throw new JsonParseException(
                                String.format("Fail to deserialize at field: %s.", fieldName));
                    }
                    jp.nextToken();
                    cnt++;
                } else {
                    skipToNextField(jp);
                }
            }
            if (cnt < arity && failOnMissingField) {
                throw new JsonParseException("Some field is missing in the JSON data.");
            }
            return row;
        };
    }

    public static void skipToNextField(JsonParser jp) throws IOException {
        switch (jp.currentToken()) {
            case START_OBJECT:
            case START_ARRAY:
                int match = 1;
                JsonToken token;
                while (match > 0) {
                    token = jp.nextToken();
                    if (token == JsonToken.END_ARRAY || token == JsonToken.END_OBJECT) {
                        match--;
                    } else if (token == JsonToken.START_ARRAY || token == JsonToken.START_OBJECT) {
                        match++;
                    }

View on GitHub (pinned to 2f3c205e92)

Solutions

Set 'json.fail-on-missing-field'='false' (absent fields become null) if absence is legitimate
Make the producer always emit all declared fields (explicit nulls)
Keep the flag true only when the contract really requires every field

Example fix

-- before
'json.fail-on-missing-field' = 'true'

-- after
'json.fail-on-missing-field' = 'false'

Defensive patterns

Strategy: validation

Validate before calling

// policy check before enabling: can every declared column be absent in real data?
// if yes -> do not enable fail-on-missing-field
if (optionalFieldsExist(rowType)) {
    Preconditions.checkArgument(!failOnMissingField, "optional columns present; disable fail-on-missing-field");
}

Try / catch

catch (JsonParseException e) {
    if (e.getMessage().contains("field is missing")) {
        // either fix producer to emit all fields, or disable the option
    }
}

Prevention

Only enable fail-on-missing-field when the producer contract guarantees all keys
Prefer explicit nulls from producers over absent keys for nullable columns

When it happens

Trigger: failOnMissingField=true and a record omits one or more declared columns (absent key, not null-valued — explicit nulls count as present).

Common situations: Optional fields modeled as required columns; upstream schema evolution dropping fields; turning on the flag for tolerance of nulls but expecting absent keys to be null too.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/b0ff80d940c8a19a. Report an issue: GitHub.