thanos-io/thanos · error

open block

Error message

open block %s

What it means

processDownsampling wraps tsdb.OpenBlock failures as errors.Wrapf(err, "open block %s", m.ULID). The downloaded block directory cannot be opened as a TSDB block — typically missing/corrupt index, chunks, or meta.json files on local disk.

Solutions

1. Clear the block's directory under --data-dir and let the next cycle re-download it (local cache is safe to delete).
2. Check disk health and that no cleaner process removes files from --data-dir while downsampling runs.
3. Raise the process file-descriptor limit (ulimit -n) if errors indicate too many open files.
4. If the source block in the bucket is itself broken, remove it after verification so retries stop.

// validate local copy before opening
for _, f := range []string{"meta.json", "index"} {
    if _, err := os.Stat(filepath.Join(bdir, f)); err != nil {
        os.RemoveAll(bdir) // force re-download
    }
}

b, err := tsdb.OpenBlock(logger, bdir, pool, nil)
if err != nil {
    os.RemoveAll(bdir) // clear local cache and retry next cycle
    return fmt.Errorf("open block %s: %w", m.ULID, err)
}

When it happens

Trigger: tsdb.OpenBlock(bdir, pool, nil) fails because index file is unreadable/corrupt, chunk files are missing, meta.json is invalid, or the block directory was deleted mid-operation.

Common situations: Incomplete download that passed earlier checks but is missing chunk segments; disk corruption or cleanup daemon removing files from --data-dir; running out of file descriptors; checksum-identical but truncated files after a crash.

Understand the failure class

Background: "open() failed", "failed to open file", "cannot create file" — what a file open error means and how to fix it — this error's family across 42 libraries.

Related errors

• remove storage lock files
• read dir
• stat
• open index reader for block %+v
• open chunk reader for block %+v

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/fce22d6cc0b83f9d. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/downsample.go:381

	}
	level.Info(logger).Log("msg", "downloaded block", "id", m.ULID, "duration", time.Since(begin), "duration_ms", time.Since(begin).Milliseconds())

	if err := block.VerifyIndex(ctx, logger, filepath.Join(bdir, block.IndexFilename), m.MinTime, m.MaxTime); err != nil && !acceptMalformedIndex {
		return errors.Wrap(err, "input block index not valid")
	}

	begin = time.Now()

	var pool chunkenc.Pool
	if m.Thanos.Downsample.Resolution == 0 {
		pool = chunkenc.NewPool()
	} else {
		pool = downsample.NewPool()
	}

	b, err := tsdb.OpenBlock(logutil.GoKitLogToSlog(logger), bdir, pool, nil)
	if err != nil {
		return errors.Wrapf(err, "open block %s", m.ULID)
	}
	defer runutil.CloseWithLogOnErr(log.With(logger, "outcome", "potential left mmap file handlers left"), b, "tsdb reader")

	id, err := downsample.Downsample(ctx, logger, m, b, dir, resolution)
	if err != nil {
		return errors.Wrapf(err, "downsample block %s to window %d", m.ULID, resolution)
	}
	resdir := filepath.Join(dir, id.String())

	downsampleDuration := time.Since(begin)
	level.Info(logger).Log("msg", "downsampled block",
		"from", m.ULID, "to", id, "duration", downsampleDuration, "duration_ms", downsampleDuration.Milliseconds())
	metrics.downsampleDuration.WithLabelValues(m.Thanos.ResolutionString()).Observe(downsampleDuration.Seconds())

	stats, err := block.GatherIndexHealthStats(ctx, logger, filepath.Join(resdir, block.IndexFilename), m.MinTime, m.MaxTime)
	if err == nil {
		err = stats.AnyErr()
	}

View on GitHub (pinned to 35b8b99117)