{"record":{"id":"df765d9000fc22dc","repo":"firecrawl/firecrawl","slug":"schema-ref-resolution-limit-exceeded","errorCode":null,"errorMessage":"Schema $ref resolution limit exceeded","messagePattern":"Schema \\$ref resolution limit exceeded","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"apps/api/src/lib/extract/helpers/dereference-schema.ts","lineNumber":56,"sourceCode":"  function walk(node: any, activeRefs: Set<string>): any {\n    if (Array.isArray(node)) {\n      return node.map(item => walk(item, activeRefs));\n    }\n    if (node === null || typeof node !== \"object\") {\n      return node;\n    }\n    if (isRefObject(node)) {\n      const ref = node.$ref;\n      // External ref, cycle, or unresolvable pointer: leave the node as-is.\n      if (!ref.startsWith(\"#\") || activeRefs.has(ref)) {\n        return { ...node };\n      }\n      const target = resolveJsonPointer(root, ref);\n      if (target === undefined) {\n        return { ...node };\n      }\n      if (++resolutions > MAX_REF_RESOLUTIONS) {\n        throw new Error(\"Schema $ref resolution limit exceeded\");\n      }\n      const nextActive = new Set(activeRefs);\n      nextActive.add(ref);\n      return walk(target, nextActive);\n    }\n    const result: Record<string, any> = {};\n    for (const key of Object.keys(node)) {\n      result[key] = walk(node[key], activeRefs);\n    }\n    return result;\n  }\n\n  try {\n    return walk(root, new Set());\n  } catch (error) {\n    console.error(\"Failed to dereference schema:\", error);\n    throw error;\n  }","sourceCodeStart":38,"sourceCodeEnd":74,"githubUrl":"https://github.com/firecrawl/firecrawl/blob/656bffcc2883f1af5befe38766b1ff5f0469993a/apps/api/src/lib/extract/helpers/dereference-schema.ts#L38-L74","documentation":"dereferenceSchema walks the JSON schema and inlines every $ref it can resolve against the document root, counting each resolution. A hard cap of MAX_REF_RESOLUTIONS (50,000) protects against pathologically large or exponential-blowup schemas. Once the counter exceeds the cap the walk aborts with this error rather than consuming unbounded memory/CPU.","triggerScenarios":"A submitted schema whose $ref graph, once inlined, expands past 50,000 resolutions. Common shapes: deeply nested schemas, many distinct pointers into a large shared definitions block, schemas that fan out combinatorially, or schemas authored with thousands of repeated refs.","commonSituations":"Large OpenAPI/JSON-Schema documents auto-converted into an extract schema; a user-supplied schema generated by a tool that emits one $ref per field; a schema with deeply recursive type definitions that, while cycle-broken, still expands hugely.","solutions":["Simplify the schema: inline the most-referenced definitions manually and remove unused branches before passing it to extraction.","Run dereferenceSchema in isolation on the suspect schema and log the resolution count to find the offending sub-tree.","Split a monolithic schema into several smaller extract calls.","If the cap is genuinely too low for a legitimate schema, raise MAX_REF_RESOLUTIONS deliberately (it is a module-level const) after confirming memory/CPU headroom."],"exampleFix":"// before\nconst deref = await dereferenceSchema(hugeOpenApiSchema);\n\n// after\n// pre-trim to only the definitions actually referenced by the extract schema\nconst trimmed = pickDefinitions(hugeOpenApiSchema, usedRefs);\nconst deref = await dereferenceSchema(trimmed);","handlingStrategy":"validation","validationCode":"function countRefs(schema: any, seen = new Set()): number {\n  if (!schema || typeof schema !== \"object\") return 0;\n  let n = 0;\n  for (const k of Object.keys(schema)) {\n    if (k === \"$ref\" && typeof schema[k] === \"string\") n++;\n    n += countRefs(schema[k]);\n  }\n  return n;\n}\n\nconst refCount = countRefs(schema);\nif (refCount > 50_000) {\n  throw new Error(`Schema has ${refCount} $ref nodes; simplify before dereference.`);\n}","typeGuard":"function isResolvableSchema(schema: any): boolean {\n  return schema !== null && typeof schema === \"object\" && !Array.isArray(schema);\n}","tryCatchPattern":"try {\n  const deref = await dereferenceSchema(schema);\n} catch (e) {\n  if (e instanceof Error && e.message === \"Schema $ref resolution limit exceeded\") {\n    // simplify and retry, or reject the schema as too complex\n    throw new Error(\"Schema too complex to dereference inline; reduce the number of $refs.\");\n  }\n  throw e;\n}","preventionTips":["Trim schemas to only the definitions actually referenced before dereference.","Add an approximate ref-count pre-check to fail fast with a clearer message than the hard cap.","Monitor dereference latency for incoming schemas; rising latency often precedes hitting the cap."],"tags":["schema","json-schema","dereference","resource-limit","defensive-guard"],"analyzedSha":"656bffcc2883f1af5befe38766b1ff5f0469993a","analyzedAt":"2026-08-12T01:18:00.488Z","schemaVersion":2},"datasetVersion":"2026-08-12T06:17:24.410Z"}