{"record":{"id":"06ddf8be5ed5118b","repo":"thanos-io/thanos","slug":"iterate-bucket-for-parquet-metadata","errorCode":null,"errorMessage":"iterate bucket for parquet metadata","messagePattern":"iterate bucket for parquet metadata","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"pkg/block/fetcher.go","lineNumber":1138,"sourceCode":"\t}\n\n\teg.Go(func() error {\n\t\tdefer close(ch)\n\n\t\treturn f.bkt.Iter(ctx, \"\", func(name string) error {\n\t\t\tif path.Base(name) == parquetMetaFileName {\n\t\t\t\tselect {\n\t\t\t\tcase ch <- name:\n\t\t\t\tcase <-ctx.Done():\n\t\t\t\t\treturn ctx.Err()\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn nil\n\t\t}, objstore.WithRecursiveIter())\n\t})\n\n\tif err := eg.Wait(); err != nil {\n\t\treturn errors.Wrap(err, \"iterate bucket for parquet metadata\")\n\t}\n\n\tif len(merrs) > 0 {\n\t\treturn errors.Wrap(merrs.Err(), \"read parquet metadata files\")\n\t}\n\n\t// NOTE(GiedriusS): we need to be extra careful here because Parquet converter trims chunks exactly to the day's range.\n\t// The input to the Parquet converter might not be aligned exactly so we NEED to have some overlap.\n\t// So, only delete metas if the migrated blocks cover each day *fully*.\n\tfor id := range filterMigratedBlocksByDayCoverage(allMigratedBlocks, metas) {\n\t\tdelete(metas, id)\n\t\tsynced.WithLabelValues(\"parquet-converted\").Inc()\n\t}\n\n\treturn nil\n}\n\nfunc filterMigratedBlocksByDayCoverage(migratedBlocks map[ulid.ULID]struct{}, metas map[ulid.ULID]*metadata.Meta) map[ulid.ULID]struct{} {","sourceCodeStart":1120,"sourceCodeEnd":1156,"githubUrl":"https://github.com/thanos-io/thanos/blob/35b8b991177def87ed52dcf10f9b6d87f07282c8/pkg/block/fetcher.go#L1120-L1156","documentation":"Returned when the errgroup waiting on the concurrent bucket iteration for parquet metadata files fails. It wraps the first error from the iter WithReportingBkt iteration callback (e.g. listing errors from the object store).","triggerScenarios":"The objstore.Iter (with WithRecursiveIter) over the parquet metadata prefix returns an error while listing objects — bucket API failures, context cancellation, or a wrapped readMigratedBlocksFromParquetMetadata error propagated via the errgroup (the per-file read errors themselves are collected separately).","commonSituations":"Bucket unavailable or throttled during listing; IAM lacking ListObjects permission; context canceled (shutdown/deadline) mid-iteration; misconfigured bucket endpoint.","solutions":["Look at the wrapped inner error to distinguish a listing error from a read error (read errors surface separately via the next wrap)","Check bucket List permissions and endpoint/credentials config","Retry; transient object-store failures typically resolve","Ensure the context is not being canceled prematurely (timeouts, shutdown ordering)"],"exampleFix":null,"handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"if err := scanParquetMetadata(ctx); err != nil {\n    if errors.Is(err, context.Canceled) { return err }\n    return retry.WithBackoff(ctx, scanParquetMetadata, 3)\n}","preventionTips":["Verify ListObjects permission on the prefix","Keep contexts alive long enough for full bucket iteration","Check bucket endpoint/credentials config early at startup","Retry transient listing failures with backoff"],"tags":["object-storage","bucket-iteration","parquet"],"backgroundTag":"network-request-failed","analyzedSha":"35b8b991177def87ed52dcf10f9b6d87f07282c8","analyzedAt":"2026-09-07T01:49:59.689Z","contentChangedAt":"2026-09-07T01:49:59.689Z","schemaVersion":2},"datasetVersion":"2026-09-15T23:17:13.987Z"}