apache/flink · error · ValidationException

fail-on-missing-field and ignore-parse-errors shouldn't both

Error message

fail-on-missing-field and ignore-parse-errors shouldn't both be true.

What it means

ValidationException from JsonFormatOptionsUtil.validateDecodingFormatOptions, raised at table-creation time when both 'json.ignore-parse-errors' and 'json.fail-on-missing-field' are true. They encode opposite strictness policies, so the format refuses the DDL. This is the early, user-facing variant of the constructor check in AbstractJsonDeserializationSchema.

Source

Thrown at flink-formats/flink-json/src/main/java/org/apache/flink/formats/json/JsonFormatOptionsUtil.java:108

                return JsonFormatOptions.MapNullKeyMode.LITERAL;
            default:
                throw new TableException(
                        String.format(
                                "Unsupported map null key handling mode '%s'. Validator should have checked that.",
                                mapNullKeyMode));
        }
    }

    // --------------------------------------------------------------------------------------------
    // Validation
    // --------------------------------------------------------------------------------------------

    /** Validator for json decoding format. */
    public static void validateDecodingFormatOptions(ReadableConfig tableOptions) {
        boolean failOnMissingField = tableOptions.get(FAIL_ON_MISSING_FIELD);
        boolean ignoreParseErrors = tableOptions.get(IGNORE_PARSE_ERRORS);
        if (ignoreParseErrors && failOnMissingField) {
            throw new ValidationException(
                    FAIL_ON_MISSING_FIELD.key()
                            + " and "
                            + IGNORE_PARSE_ERRORS.key()
                            + " shouldn't both be true.");
        }
        validateTimestampFormat(tableOptions);
    }

    /** Validator for json encoding format. */
    public static void validateEncodingFormatOptions(ReadableConfig tableOptions) {
        // validator for {@link MAP_NULL_KEY_MODE}
        Set<String> nullKeyModes =
                Arrays.stream(JsonFormatOptions.MapNullKeyMode.values())
                        .map(Objects::toString)
                        .collect(Collectors.toSet());
        if (!nullKeyModes.contains(tableOptions.get(MAP_NULL_KEY_MODE).toUpperCase())) {
            throw new ValidationException(
                    String.format(

View on GitHub (pinned to 2f3c205e92)

Solutions

If data quality matters more than uptime: drop 'json.ignore-parse-errors' (or set false) and keep fail-on-missing-field=true
If uptime matters more: set 'json.fail-on-missing-field'='false' and keep ignore-parse-errors=true
Re-create the table after fixing the options and verify with SHOW CREATE TABLE

Example fix

-- before
WITH (
  'format'='json',
  'json.ignore-parse-errors'='true',
  'json.fail-on-missing-field'='true'
)

-- after
WITH (
  'format'='json',
  'json.ignore-parse-errors'='true',
  'json.fail-on-missing-field'='false'
)

Defensive patterns

Strategy: validation

Validate before calling

JsonFormatOptionsUtil.validateDecodingFormatOptions(tableOptions); // throws early with option keys in the message

Prevention

Decide the strictness policy once (strict vs tolerant) and set exactly one flag
Codify WITH-clause templates so conflicting pairs cannot be combined

When it happens

Trigger: CREATE TABLE ... WITH ('format'='json', 'json.ignore-parse-errors'='true', 'json.fail-on-missing-field'='true'); also programmatic discovery with both flags enabled.

Common situations: Copy-pasted WITH clauses hardened for both dirty and sparse data; enabling ignore-parse-errors to survive dirty feeds while leaving fail-on-missing-field from an earlier strict setup.

Related errors

AI-assisted analysis of apache/flink@2f3c205e92 (2026-08-14). Data as JSON: /api/errors/aed01234593c4eba. Report an issue: GitHub.