What causes "Some field is missing in the Json data." in apache/flink?

Set 'json.fail-on-missing-field'='false' so absent nested fields deserialize as null

apache/flink · error · org.apache.flink.formats.json.JsonParseException

Some field is missing in the Json data.

Error message

Some field is missing in the Json data.

What it means

JsonParseException from RowNestedConverter.convertNotNull: the nested-projection variant of the missing-field check. After parsing the object, fewer projected fields were found than expected while 'json.fail-on-missing-field'='true'. It applies to the nested fields selected via projection, not the full row.

Source

Thrown at flink-formats/flink-json/src/main/java/org/apache/flink/formats/json/JsonParserToRowDataConverters.java:630

            jp.nextToken();
            while (jp.currentToken() != JsonToken.END_OBJECT) {
                if (cnt >= arity) {
                    skipToNextField(jp);
                    continue;
                }
                String fieldName = jp.getText();
                jp.nextToken();
                ProjectedConverter converter = fieldConverters.get(fieldName);
                if (converter != null) {
                    converter.convert(jp, outputRow);
                    jp.nextToken();
                    cnt++;
                } else {
                    skipToNextField(jp);
                }
            }
            if (cnt < arity && failOnMissingField) {
                throw new JsonParseException("Some field is missing in the Json data.");
            }
        }
    }
}

View on GitHub (pinned to 2f3c205e92)

Solutions

Set 'json.fail-on-missing-field'='false' so absent nested fields deserialize as null
Have the producer always emit the nested keys (explicit nulls)
Restrict projections to nested fields guaranteed present, or handle nulls downstream

Example fix

-- before
'json.fail-on-missing-field' = 'true'

-- after
'json.fail-on-missing-field' = 'false'

Defensive patterns

Strategy: validation

Validate before calling

// before enabling fail-on-missing-field with nested projections:
// verify every projected nested key exists in representative records
for (String sample : samples) {
    JsonNode root = mapper.readTree(sample);
    for (String path : projectedNestedPaths) {
        Preconditions.checkState(resolve(root, path) != null, "missing " + path);
    }
}

Try / catch

catch (JsonParseException e) {
    if (e.getMessage().contains("field is missing")) {
        // absent nested key with fail-on-missing-field=true; disable option or fix producer
    }
}

Prevention

Nested optional fields + fail-on-missing-field is the classic trigger; pick one
Have producers emit null-valued nested keys instead of omitting them

When it happens

Trigger: failOnMissingField=true, a projection over nested ROW fields, and a record where one of those nested keys is absent from its JSON object.

Common situations: Optional nested attributes modeled as required; producers occasionally omitting empty sub-objects; enabling fail-on-missing-field while migrating schemas where nested keys come and go.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/5ed7c7bc82e6aa8f. Report an issue: GitHub.