thanos-io/thanos · critical

create default tenant data dir

Error message

create default tenant data dir: %v

What it means

Received fails to create the data directory for the default tenant (hashrings without an explicit tenant fall back to the default tenant, defaultTenantID "default-tenant"). This wraps the error returned by dataDir.MkdirAll when the per-tenant TSDB data directory cannot be created. It is fatal for startup because the default tenant must be able to store series.

Solutions

1. Check that the --tsdb.path directory exists and is writable by the receive process user (ls -ld, or chown/chmod it).
2. Verify the mount is not read-only and the filesystem has free space (mount | grep, df -h).
3. If a non-directory exists at the path, remove or rename it and restart.
4. Run with correct security context (fsGroup/SELinux labels) in Kubernetes or as a user with write access.

Example fix

// before
mkdir -p /var/lib/thanos/receive && thanos receive --tsdb.path=/var/lib/thanos/receive  # dir owned by root
// after
sudo chown thanos:thanos /var/lib/thanos/receive && chmod 750 /var/lib/thanos/receive

import os, stat
def ensure_tenant_data_dir(base, tenant="default-tenant"):
    p = os.path.join(base, tenant)
    if not os.path.isdir(base):
        raise SystemExit(f"tsdb path {base} does not exist")
    if not os.access(base, os.W_OK | os.X_OK):
        raise SystemExit(f"tsdb path {base} not writable by uid={os.getuid()}")
    if os.path.exists(p) and not os.path.isdir(p):
        raise SystemExit(f"{p} exists but is not a directory")
    return p

def is_writable_dir(path):
    return os.path.isdir(path) and os.access(path, os.W_OK)

When it happens

Trigger: dataDir.MkdirAll(defaultTenantID, 0750) fails: the parent --tsdb.path does not exist or is not writable, the filesystem is read-only or full, or an OS-level permission/SELinux issue blocks mkdir.

Common situations: Running the receive container as a non-root user with a hostPath/PV owned by root; the data directory was mounted read-only; disk full on the node; a file exists where the directory is expected.

Understand the failure class

Background: mkdir permission denied (EACCES): failed to create directory errors explained — this error's family across 32 libraries.

Related errors

• create working compact directory
• create working downsample directory
• remove storage lock files
• create meta fetcher
• create dir

AI-assisted analysis of thanos-io/thanos@35b8b99117 (2026-09-07). Data as JSON: /api/errors/9122ea9545810d3d. Report an issue: GitHub.

Appendix: source

Thrown at cmd/thanos/receive.go:866

			}, func(error) {
				cancel()
			})
		}
	}

	return nil
}

func createDefautTenantTSDB(logger log.Logger, defaultTenantID string, dataDir *os.Root) error {
	if _, err := dataDir.Stat(defaultTenantID); !os.IsNotExist(err) {
		level.Info(logger).Log("msg", "default tenant data dir already present, will not create")
		return nil
	}

	level.Info(logger).Log("msg", "default tenant data dir not found, creating", "defaultTenantID", defaultTenantID)

	if err := dataDir.MkdirAll(defaultTenantID, 0750); err != nil {
		return errors.Wrapf(err, "create default tenant data dir: %v", path.Join(dataDir.Name(), defaultTenantID))
	}

	return nil
}

type receiveConfig struct {
	httpBindAddr    *string
	httpGracePeriod *model.Duration
	httpTLSConfig   *string

	grpcConfig grpcConfig

	replicationAddr       string
	rwAddress             string
	rwServerCert          string
	rwServerKey           string
	rwServerClientCA      string
	rwClientCert          string

View on GitHub (pinned to 35b8b99117)