{"record":{"id":"15d9ec245b6712d2","repo":"Hmbown/CodeWhale","slug":"file-exceeds-the-64-mib-mvp-import-limit-split-the-export-by","errorCode":null,"errorMessage":"File exceeds the 64 MiB MVP import limit. Split the export by trace.","messagePattern":"File exceeds the 64 MiB MVP import limit\\. Split the export by trace\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"pet/src/core/ingest.ts","lineNumber":224,"sourceCode":"      const ea = attributes(record.attributes), time = Number(ns(record.timeUnixNano, 'event.timeUnixNano') - origin) / 1e6;\n      const ename = String(record.name ?? 'span event');\n      events.push({ schemaVersion: 1, id: `${s.spanId}/event/${i}`, traceId: s.traceId, parentId: s.spanId,\n        startTime: time, endTime: time, agentId: e.agentId, name: ename, category: categoryFor(ename, ea),\n        status: ename === 'exception' ? 'error' : 'unknown', attributes: ea, raw: { event: record, spanId: s.spanId, resource: rec.resource, scope: rec.scope } });\n    }\n    if (events.length > maxEvents) throw new Error(`Import exceeds the ${maxEvents.toLocaleString()} event limit, including span events.`);\n  }\n  return { events, origins: new Map([...bases].map(([k, v]) => [k, v.toString()])), warnings };\n}\n\n/** Parse strictly: malformed lines or duplicate identities never disappear silently. */\nexport function importTrace(text: string, filename = 'Imported trace', options: ImportOptions = {}): Trace[] {\n  const mode = options.privacy ?? 'redact', maxEvents = options.maxEvents ?? 250_000;\n  if (!['redact', 'metadata', 'retain'].includes(mode)) throw new Error('Unknown privacy mode.');\n  const maxTraces = options.maxTraces ?? 8;\n  if (!Number.isInteger(maxEvents) || maxEvents < 1 || maxEvents > 250_000) throw new Error('maxEvents must be in [1, 250000].');\n  if (!Number.isInteger(maxTraces) || maxTraces < 1 || maxTraces > 64) throw new Error('maxTraces must be in [1, 64].');\n  if (new TextEncoder().encode(text).length > (options.maxBytes ?? 64 * 1024 * 1024)) throw new Error('File exceeds the 64 MiB MVP import limit. Split the export by trace.');\n  const trimmed = text.replace(/^\\uFEFF/, '').trim();\n  if (!trimmed) throw new Error('The trace file is empty.');\n  let document: unknown;\n  try { document = JSON.parse(trimmed); }\n  catch {\n    document = trimmed.split(/\\r?\\n/).filter(l => l.trim()).map((line, i) => {\n      try { return JSON.parse(line); } catch { throw new Error(`Invalid JSON on nonempty line ${i + 1}. Import cancelled; no rows were skipped.`); }\n    });\n  }\n  // Transform before both normalization and raw retention, so raw cannot bypass redaction.\n  const safe = mode === 'retain' ? (options.redactor ? options.redactor(document, '') : document) : redact(document, '', options.redactor);\n  const root = obj(safe);\n  if(root.format === 'whalesong.evidence/v1') return [evidenceToTrace(validateBundle(root, Math.min(maxEvents, 100_000)))];\n  if (isCodewhaleSession(safe)) {\n    const trace = fromCodewhaleSession(safe, filename, maxEvents);\n    trace.privacy = mode;\n    trace.events = trace.events.map(e => privacyEvent(e, mode));\n    return [trace];","sourceCodeStart":206,"sourceCodeEnd":242,"githubUrl":"https://github.com/Hmbown/CodeWhale/blob/433685b2024e7bc4c99e1e2e326bcad39b4d9d65/pet/src/core/ingest.ts#L206-L242","documentation":"importTrace enforces a hard 64 MiB limit on the raw input text (measured in UTF-8 bytes), overridable only downward via options.maxBytes. Oversized files are rejected before parsing so a huge export cannot exhaust memory in the browser/runtime; the message tells you to split the export per trace.","triggerScenarios":"Passing a JSON or JSONL trace file whose UTF-8 byte length exceeds 64 MiB (or the smaller options.maxBytes if set) to importTrace.","commonSituations":"Long-running services exporting days of OTLP spans into one file; cumulative exports that kept growing; an options.maxBytes override set smaller than the actual file.","solutions":["Split the export into multiple files by trace ID and import each separately.","Raise or re-check the limit only if you set options.maxBytes yourself; the built-in 64 MiB cap is not configurable upward.","Reduce imported data at the source (sampling, shorter time window, fewer attributes) before export.","Pre-filter the text before calling importTrace to remove unneeded traces."],"exampleFix":"// before\nimportTrace(hugeText); // > 64 MiB\n// after\nconst chunks = splitByTrace(hugeText); // split export per trace ID\nchunks.forEach(c => importTrace(c));","handlingStrategy":"validation","validationCode":"if (new TextEncoder().encode(text).length > 64 * 1024 * 1024) {\n  throw new Error('File too large: split by trace before importing.');\n}","typeGuard":null,"tryCatchPattern":"try {\n  importTrace(text);\n} catch (e) {\n  if (e instanceof Error && e.message.includes('64 MiB')) {\n    console.error('Split the export by trace ID and import each part.');\n  }\n}","preventionTips":["Check file size (bytes) before reading/importing.","Export per trace or per time slice instead of one giant file.","Remember the limit is UTF-8 byte length, not character count.","Set options.maxBytes only when you want a stricter cap; it cannot raise 64 MiB."],"tags":["validation","file-size","ingest"],"backgroundTag":"file-size-limit-exceeded","analyzedSha":"433685b2024e7bc4c99e1e2e326bcad39b4d9d65","analyzedAt":"2026-09-15T12:24:24.634Z","contentChangedAt":"2026-09-15T12:24:24.634Z","schemaVersion":2},"datasetVersion":"2026-09-22T11:17:16.035Z"}