litedb-org/LiteDB · warning · LiteException

0

0

Error message

OrderBy/GroupBy operation are supported only in virtual collection with less than {VIRTUAL_INDEX_MAX_CACHE} documents

What it means

Thrown by IndexVirtual.Load(PageAddress) when an OrderBy or GroupBy operation requires document lookup from a virtual (in-memory/external) collection, but the internal cache has been nulled because the source exceeded VIRTUAL_INDEX_MAX_CACHE (2000) documents. IndexVirtual caches source documents by rawId up to 2000 entries to support OrderBy/GroupBy lookups; past that limit the cache is dropped to bound memory, and any subsequent lookup that needs it fails.

Solutions

1. Reduce the source to fewer than 2000 documents before applying OrderBy/GroupBy (filter first).
2. Perform the ordering/grouping in application code (LINQ) before/instead of in the LiteDB query.
3. If using a real (non-virtual) collection, ensure the query uses a physical index so IndexVirtual is not used.
4. Paginate the virtual source with Limit/Offset to stay under the cache threshold.

Example fix

// before — virtual source too large for ORDER BY
var docs = LoadExternalDocs(); // 5000 docs
var q = db.GetCollection("$dump").Query().OrderBy("$.name").ToEnumerable();
// throws during enumeration

// after — sort in app, or filter/paginate first
var sorted = docs.OrderBy(d => d["name"]).ToList();
// or paginate: .Limit(2000) before OrderBy

const int VIRTUAL_MAX = 2000;

public List<BsonDocument> QueryExternal(LiteDatabase db, IEnumerable<BsonDocument> source)
{
    var materialized = source.Take(VIRTUAL_MAX + 1).ToList();
    if (materialized.Count > VIRTUAL_MAX)
    {
        // Source is too large for in-DB OrderBy/GroupBy — sort in app code instead.
        materialized = source.OrderBy(d => d["name"].AsString).ToList();
        return materialized;
    }
    // Safe to use virtual collection with OrderBy.
    return db.GetCollection("$dump").InsertBulk(materialized) > 0
        ? db.GetCollection("$dump").Query().OrderBy("$.name").ToList()
        : materialized;
}

static bool IsSafeForVirtualOrderBy(IEnumerable<BsonDocument> source)
{
    // Only safe if you can cheaply count; otherwise cap with Take.
    return source.Take(2001).Count() <= 2000;
}

try
{
    var results = virtualQuery.OrderBy("$.name").ToList();
}
catch (LiteException ex) when (ex.Message.Contains("virtual collection with less than"))
{
    // Fall back to in-memory LINQ ordering over a materialized list.
    results = source.OrderBy(d => d["name"].AsString).ToList();
}

When it happens

Trigger: Running a query with .OrderBy() or .GroupBy() over an external/virtual source (e.g. a system collection, a $dump, or a manually-supplied IEnumerable<BsonDocument> in IndexVirtual) that yields more than 2000 documents. The cache is discarded after the 2000th document; a later OrderBy/GroupBy phase triggers Load(PageAddress) and throws.

Common situations: Sorting or grouping large in-memory result sets passed as an external source; querying system collections ($dump, $sequences) with ORDER BY over more than 2000 rows; using ILiteCollection.Query over a virtual source.

Related errors

• GROUP BY expression do not support INCLUDE
• 0
• 0
• 0
• 0

AI-assisted analysis of litedb-org/LiteDB@f906a5f850 (2026-08-13). Data as JSON: /api/errors/0e890d593b2bbb21. Report an issue: GitHub.

Appendix: source

Thrown at LiteDB/Engine/Query/IndexQuery/IndexVirtual.cs:64

                {
                    _cache[rawId] = doc;

                    if (_cache.Count > VIRTUAL_INDEX_MAX_CACHE) _cache = null;
                }

                // return an fake indexNode
                yield return new IndexNode(doc);
            }
        }

        public BsonDocument Load(IndexNode node)
        {
            return node.Key as BsonDocument;
        }

        public BsonDocument Load(PageAddress rawId)
        {
            if (_cache == null) throw new LiteException(0, $"OrderBy/GroupBy operation are supported only in virtual collection with less than {VIRTUAL_INDEX_MAX_CACHE} documents");

            return _cache[rawId.PageID];
        }

        public override string ToString()
        {
            return string.Format("FULL COLLECTION SCAN");
        }
    }
}

View on GitHub (pinned to f906a5f850)