{"record":{"id":"e591e63afd779fc1","repo":"thanos-io/thanos","slug":"meta-json-v-unmarshal-v","errorCode":null,"errorMessage":"meta.json %v unmarshal: %v","messagePattern":"meta\\.json (.+?) unmarshal: (.+?)","errorType":"exception","errorClass":"ErrorSyncMetaCorrupted","httpStatus":null,"severity":"error","filePath":"pkg/block/fetcher.go","lineNumber":496,"sourceCode":"\tr, err := f.bkt.ReaderWithExpectedErrs(f.bkt.IsObjNotFoundErr).Get(ctx, metaFile)\n\tif f.bkt.IsObjNotFoundErr(err) {\n\t\t// Meta.json was deleted between bkt.Exists and here.\n\t\treturn nil, errors.Wrapf(ErrorSyncMetaNotFound, \"%v\", err)\n\t}\n\tif err != nil {\n\t\treturn nil, errors.Wrapf(err, \"get meta file: %v\", metaFile)\n\t}\n\n\tdefer runutil.CloseWithLogOnErr(f.logger, r, \"close bkt meta get\")\n\n\tmetaContent, err := io.ReadAll(r)\n\tif err != nil {\n\t\treturn nil, errors.Wrapf(err, \"read meta file: %v\", metaFile)\n\t}\n\n\tm := &metadata.Meta{}\n\tif err := json.Unmarshal(metaContent, m); err != nil {\n\t\treturn nil, errors.Wrapf(ErrorSyncMetaCorrupted, \"meta.json %v unmarshal: %v\", metaFile, err)\n\t}\n\n\tif m.Version != metadata.TSDBVersion1 {\n\t\treturn nil, errors.Errorf(\"unexpected meta file: %s version: %d\", metaFile, m.Version)\n\t}\n\n\t// Best effort cache in local dir.\n\tif f.cacheDir != \"\" {\n\t\tif err := os.MkdirAll(cachedBlockDir, os.ModePerm); err != nil {\n\t\t\tlevel.Warn(f.logger).Log(\"msg\", \"best effort mkdir of the meta.json block dir failed; ignoring\", \"dir\", cachedBlockDir, \"err\", err)\n\t\t}\n\n\t\tif err := m.WriteToDir(f.logger, cachedBlockDir); err != nil {\n\t\t\tlevel.Warn(f.logger).Log(\"msg\", \"best effort save of the meta.json to local dir failed; ignoring\", \"dir\", cachedBlockDir, \"err\", err)\n\t\t}\n\t}\n\treturn m, nil\n}","sourceCodeStart":478,"sourceCodeEnd":514,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/fetcher.go#L478-L514","documentation":"loadMeta in BaseFetcher reads a block's meta.json from the object store and unmarshals it into metadata.Meta. When json.Unmarshal fails, the error is wrapped with ErrorSyncMetaCorrupted and this message. It means the block's meta.json exists but is not valid JSON (or not valid for the Meta struct), so the block is treated as corrupted during bucket sync.","triggerScenarios":"Calling fetchMeta/fetchMetadata (bucket sync) on a block whose meta.json is truncated, empty, non-JSON bytes, or has fields with wrong JSON types (e.g. version as string) so json.Unmarshal into *metadata.Meta fails.","commonSituations":"Interrupted block upload leaving a partial meta.json; manual editing of meta.json in the bucket; an older/other Thanos or Prometheus version writing a schema json.Unmarshal cannot accept; corrupted object-store objects (misconfigured storage returning garbage).","solutions":["Check the raw meta.json of the failing block in the bucket (download it and run it through a JSON validator); re-upload or re-truncate the block if corrupted.","Delete the corrupted block's meta.json or the partial block so sync treats it as partial rather than corrupted, then re-sync.","Verify the object store configuration/gateway isn't returning truncated objects (checksums, network).","Re-create the block by re-uploading from the source Prometheus/Compactor instead of hand-repairing meta.json."],"exampleFix":"// before: block meta.json truncated in bucket\n{\"ulid\": \"01ARZ3NDEKTSV4RRFFQ69G5FAV\", \"minTime\": 1577836800000\n// after: complete valid meta.json (re-upload block)\n{\"ulid\":\"01ARZ3NDEKTSV4RRFFQ69G5FAV\",\"minTime\":1577836800000,\"maxTime\":1577923200000,\"stats\":{\"numSamples\":0},\"compaction\":{\"level\":1,\"sources\":[\"01ARZ3NDEKTSV4RRFFQ69G5FAV\"]},\"version\":1}","handlingStrategy":"try-catch","validationCode":"var raw map[string]any\nif err := json.Unmarshal(metaContent, &raw); err != nil {\n    // meta.json is not valid JSON before syncing: repair or re-upload block\n}","typeGuard":"func isCorruptedMetaError(err error) bool {\n    return errors.Is(err, block.ErrorSyncMetaCorrupted) || strings.Contains(err.Error(), \"unmarshal\")\n}","tryCatchPattern":"metas, err := fetcher.FetchMetadata(ctx, ...)\nif err != nil {\n    var corrupted []ulid.ULID\n    if errors.Is(err, block.ErrorSyncMetaCorrupted) || strings.Contains(err.Error(), \"meta.json\") {\n        // re-download meta.json, validate JSON, re-upload or exclude block, then retry sync\n    } else {\n        return err\n    }\n}","preventionTips":["Never hand-edit meta.json in the bucket; re-upload whole blocks instead.","Ensure uploads are atomic/complete before exposing block dirs to sync (Thanos uploader ordering).","Monitor noMetas/corruptedMetas sync metrics and alert on increases.","Validate object store integrity (checksums) when using non-native gateways."],"tags":["object-storage","json","blocks","sync"],"backgroundTag":"json-unmarshal-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}