{"record":{"id":"81d0935ad2005964","repo":"LykosAI/StabilityMatrix","slug":"civarchive-list-page-was-missing-pageprops","errorCode":null,"errorMessage":"CivArchive list page was missing pageProps","messagePattern":"CivArchive list page was missing pageProps","errorType":"exception","errorClass":"InvalidOperationException","httpStatus":null,"severity":"error","filePath":"StabilityMatrix.Core/Api/CivArchiveApiClient.cs","lineNumber":91,"sourceCode":"    {\n        ArgumentNullException.ThrowIfNull(filters);\n\n        // /search is the only list route without server-side fixed filters — the curated\n        // routes (/top-models, /hot-models, …) force their own period/sort and silently\n        // override whatever the query string says (e.g. /top-models pins period=quarter).\n        var routePath = string.IsNullOrWhiteSpace(filters.RoutePath) ? \"/search\" : filters.RoutePath;\n        var relativePath = BuildSearchDataPath(routePath, filters);\n\n        if (searchCache.Get(relativePath) is { } cached && cached.IsFresh(SearchCacheTtl))\n        {\n            return cached.Value;\n        }\n\n        var response = await GetNextDataAsync<CivArchiveListPageResponse>(relativePath, cancellationToken);\n\n        var pageProps =\n            response.PageProps\n            ?? throw new InvalidOperationException(\"CivArchive list page was missing pageProps\");\n\n        var effectiveFilters = pageProps.Filters?.ToSearchFilters() ?? filters;\n\n        var result = new CivArchiveSearchResponse\n        {\n            Results = pageProps.Data?.Results ?? [],\n            FilterOptions = new CivArchiveFilterOptions\n            {\n                BaseModels = pageProps.FilterOptions?.BaseModels ?? [],\n                ModelTypes = pageProps.FilterOptions?.ModelTypes ?? [],\n            },\n            EffectiveFilters = effectiveFilters,\n            CanonicalUrl = pageProps.CanonicalUrl ?? string.Empty,\n            TotalHits = pageProps.Data?.TotalHits ?? 0,\n        };\n\n        searchCache.Add(\n            relativePath,","sourceCodeStart":73,"sourceCodeEnd":109,"githubUrl":"https://github.com/LykosAI/StabilityMatrix/blob/af93d6ef57c01cd890d7e0ad0a9ea8c9fcda3002/StabilityMatrix.Core/Api/CivArchiveApiClient.cs#L73-L109","documentation":"SearchAsync in CivArchiveApiClient throws InvalidOperationException when the Next.js __NEXT_DATA__ payload fetched from a CivArchive list/search page has no PageProps property. The client relies on pageProps to carry search results and filters; its absence means the page shape changed or the request did not return a real list page.","triggerScenarios":"Calling CivArchiveApiClient.SearchAsync when the scraped /search page's __NEXT_DATA__ JSON lacks a pageProps object (page moved, site redesign, redirect/error page returned instead of the list).","commonSituations":"CivArchive website layout changes breaking the scraped contract, CDN serving an error/challenge page, network middleware returning HTML other than the expected Next.js page.","solutions":["Check the raw JSON the client fetches to see if pageProps moved or was renamed","Update CivArchiveApiClient / CivArchiveListPageResponse to match the current site schema","Retry later if the site is temporarily serving an error page","Report/track the mismatch since this is a third-party site scraping dependency"],"exampleFix":"// before\nvar pageProps = response.PageProps ?? throw new InvalidOperationException(\"CivArchive list page was missing pageProps\");\n// after\nif (response.PageProps is null) { log.Warn(\"CivArchive list page had no pageProps\"); return CivArchiveSearchResponse.Empty; }","handlingStrategy":"try-catch","validationCode":"var html = await http.GetStringAsync(searchUrl);\nbool hasPageProps = html.Contains(\"__NEXT_DATA__\") && html.Contains(\"pageProps\");","typeGuard":"bool HasPageProps(CivArchiveListPageResponse r) => r?.PageProps is not null;","tryCatchPattern":"try { return await api.SearchAsync(filters, ct); }\ncatch (InvalidOperationException) { return cachedResults ?? CivArchiveSearchResponse.Empty; }","preventionTips":["Cache successful search responses for fallback","Monitor CivArchive site layout changes","Treat scraped third-party data as best-effort","Log raw payload on failure to speed schema-fix turnaround"],"tags":["scraping","nextjs","api","upstream"],"backgroundTag":"unexpected-response-shape","analyzedSha":"af93d6ef57c01cd890d7e0ad0a9ea8c9fcda3002","analyzedAt":"2026-09-12T19:02:43.389Z","contentChangedAt":"2026-09-12T19:02:43.389Z","schemaVersion":2},"datasetVersion":"2026-09-16T04:17:20.429Z"}