pingcap/tidb · critical

Out Of Global Memory Limit!

Error message

Out Of Global Memory Limit!

What it means

Fired by the globalPanicOnExceed memory action in TiDB's executor (select.go). When the tracker labeled LabelForGlobalMemory (the server-wide memory tracker) exceeds its quota, TiDB deliberately panics the owning session/goroutine with 'Out Of Global Memory Limit!' to keep the process from being OOM-killed by the OS. The panic unwinds the query and the error surfaces to the client.

Source

Thrown at pkg/executor/select.go:182

}

// Action panics when storage usage exceeds storage quota.
func (a *globalPanicOnExceed) Action(t *memory.Tracker) {
	a.mutex.Lock()
	defer a.mutex.Unlock()
	msg := ""
	switch t.Label() {
	case memory.LabelForGlobalStorage:
		msg = globalPanicStorageExceed
	case memory.LabelForGlobalMemory:
		msg = globalPanicMemoryExceed
	case memory.LabelForGlobalAnalyzeMemory:
		msg = globalPanicAnalyzeMemoryExceed
	default:
		msg = "Out of Unknown Resource Quota!"
	}
	// TODO(hawkingrei): should return error instead.
	panic(msg)
}

// GetPriority get the priority of the Action
func (*globalPanicOnExceed) GetPriority() int64 {
	return memory.DefPanicPriority
}

// SelectLockExec represents a select lock executor.
// It is built from the "SELECT .. FOR UPDATE" or the "SELECT .. LOCK IN SHARE MODE" statement.
// For "SELECT .. FOR UPDATE" statement, it locks every row key from source Executor.
// After the execution, the keys are buffered in transaction, and will be sent to KV
// when doing commit. If there is any key already locked by another transaction,
// the transaction will rollback and retry.
type SelectLockExec struct {
	exec.BaseExecutor

	Lock *ast.SelectLockInfo
	keys []kv.Key

View on GitHub (pinned to d01f9615c1)

Solutions

1. Raise the global memory limit (or add machine memory) via memory_usage_limit / tidb_server_memory_limit so the workload fits.
2. Reduce peak per-query memory: add LIMIT, filter earlier, split the query, or create suitable indexes so large hash joins/sorts are avoided.
3. Enable/verify spill for the offending operators (e.g. tidb_enable_pseudo_for_outbound_stats off; tidb_mem_quota_query plus spill for sort/join/agg: tidb_enable_spilled_result_protection, tidb_prefer_flashback... - specifically enable disk spill for Sort/Join/Agg via their tidb_* spill variables) so they go to temp storage instead of RAM.
4. Kill or throttle the memory-heavy sessions (max_connections, resource control with RUNAWAY queries) so global usage stays under quota.
5. If usage is dominated by a single system operation (e.g. ANALYZE or GC), schedule it off-peak.

Example fix

-- before: single giant aggregation blows the global limit
SELECT c_custkey, SUM(o_totalprice) FROM orders GROUP BY c_custkey; -- panics

-- after: bound the query and let TiDB spill
SET GLOBAL tidb_mem_quota_query = 4294967296; -- per-query bound
SET SESSION tidb_enable_spilled_result = ON;  -- allow spill for sort/agg
SELECT c_custkey, SUM(o_totalprice) FROM orders WHERE o_orderdate >= '2024-01-01' GROUP BY c_custkey;

-- before running big queries, check current global memory headroom
SELECT MEMORY_OFFSET, MEMORY_USAGE, MEMORY_MAX,
       ROUND((MEMORY_USAGE / MEMORY_MAX) * 100, 1) AS used_pct
FROM information_schema.INSTANCE_MEMORY_USAGE
ORDER BY MEMORY_USAGE DESC;

-- catch at the client: the panic surfaces as a query error, so a retry wrapper
gives the query a chance once memory frees up (do NOT blind-retry hot loops)
EXECUTE stmt; -- on error 1105 'Out Of Global Memory Limit!': shed load, then retry once

When it happens

Trigger: Any query or set of concurrent queries whose combined tracked memory exceeds the global memory limit (memory_usage_limit / tidb_server_memory_limit and friends): huge joins/sorts/aggregations with no spill, wide table scans, many concurrent analytical queries, or ANALYZE under the global tracker.

Common situations: Server sized with memory_usage_limit close to physical RAM while workloads spike; spill-to-disk disabled for the operators that would otherwise spill (tidb_enable_spilled_result / spill settings for join/agg/sort); a migration from an older TiDB where only per-query mem-quota-query applied; slow concurrent stats or large DML ballooning global tracked usage.

Related errors

• Out Of Global Analyze Memory Limit!
• Out of Unknown Resource Quota!
• Out Of Quota For Local Temporary Space!
• fail to create iterator
• TiFlash does not support vector index with pedicates

AI-assisted analysis of pingcap/tidb@d01f9615c1 (2026-08-15). Data as JSON: /api/errors/aa052de0d287a69e. Report an issue: GitHub.