bytebase/bytebase · error

failed to unmarshal field %q

Error message

failed to unmarshal field %q

What it means

In a GET _source response, each top-level document field arrives as its own column value. maskElasticsearchGetSourceColumn parses each field's JSON string before walking it with the schema masker; this error means the field value string was not valid JSON.

Source

Thrown at backend/api/v1/document_masking.go:1223

}

// maskElasticsearchGetSourceColumn masks a single field value from a GET _source response.
// In _source responses, each top-level document field becomes a separate column.
// fieldName is the column name (= field name), valueJSON is the marshaled value.
func maskElasticsearchGetSourceColumn(fieldName, valueJSON string, objectSchema *storepb.ObjectSchema, semanticTypeToMasker map[string]masker.Masker) (string, error) {
	var childSchema *storepb.ObjectSchema
	if structKind := objectSchema.GetStructKind(); structKind != nil {
		if props := structKind.GetProperties(); props != nil {
			childSchema = props[fieldName]
		}
	}
	if childSchema == nil {
		return valueJSON, nil
	}

	var value any
	if err := json.Unmarshal([]byte(valueJSON), &value); err != nil {
		return "", errors.Wrapf(err, "failed to unmarshal field %q", fieldName)
	}

	masked, err := walkAndMaskJSONRecursive(value, childSchema, semanticTypeToMasker)
	if err != nil {
		return "", err
	}

	out, err := json.Marshal(masked)
	if err != nil {
		return "", errors.Wrapf(err, "failed to marshal masked field %q", fieldName)
	}
	return string(out), nil
}

// maskElasticsearchMGetSource masks _source in each doc within a _mget response.
// The docsColumnJSON is the JSON string from the "docs" column.
func maskElasticsearchMGetSource(docsColumnJSON string, objectSchema *storepb.ObjectSchema, semanticTypeToMasker map[string]masker.Masker) (string, error) {
	var docs []any

View on GitHub (pinned to 1870550677)

Solutions

1. Inspect the offending field's raw value (named in the error) to see why it isn't valid JSON.
2. Ensure the ES driver serializes every field value as a JSON string, including scalars.
3. Fix data or driver truncation for large text fields.
4. Retry after correcting the field serialization.

Example fix

// before: column value for field "note" is: some unquoted text
// after: serialize as valid JSON string: "some unquoted text"

// Before masking a _source field column:
// if !json.Valid([]byte(valueJSON)) { return valueJSON, nil }

func isJSONString(s string) bool { return json.Valid([]byte(s)) }

masked, err := maskElasticsearchGetSourceColumn(colName, value, ...)
if err != nil && strings.Contains(err.Error(), "failed to unmarshal field") {
    // pass through the original value and log the field name
}

When it happens

Trigger: MaskGetSource API where a column's value string fails json.Unmarshal — e.g. the driver returned a scalar as a raw unquoted string, or the value was truncated/mangled.

Common situations: Driver serializing primitive fields inconsistently (string vs JSON-encoded); truncation of long text fields; ES/text extraction tools quoting values differently.

Understand the failure class

Background: "failed to unmarshal" / json.Unmarshal errors: why parsing a response into a Go struct fails and how to fix it — this error's family across 23 libraries.

Related errors

• failed to marshal masked field %q
• failed to unmarshal hits column
• failed to marshal masked hits column
• failed to unmarshal docs column
• failed to marshal masked docs column

AI-assisted analysis of bytebase/bytebase@1870550677 (2026-09-06). Data as JSON: /api/errors/43cf9ed2b049f175. Report an issue: GitHub.