A WARC (Web ARChive) file is a container for captured web resources and associated metadata, standardized by ISO 28500. Unlike a spreadsheet of extracted fields, it can preserve HTTP responses and the context needed to inspect or process a capture again after the live page changes.
A WARC file concatenates records with their own headers and content blocks. Response records can contain HTTP status, response headers, and payload bytes; separate request and metadata records can describe how the resource was retrieved. Record identifiers and timestamps help connect those pieces.
This separation is useful when an extractor changes. A stored response can be parsed again to recover fields that the original extraction missed, without another request to the source. Keep extraction results linked to capture identifiers so a reported value can be traced to the bytes that produced it.
WARC supports revisit records that describe a repeat capture without storing the same payload again. Resolving a revisit can require the earlier record it references. A file containing revisit records alone may therefore be insufficient to reconstruct the content.
The common .warc.gz suffix indicates gzip compression. The specification recommends independent compression of individual records, which allows an external index to locate and decompress one record without reading every preceding record. Compression, indexing, and replay are separate concerns from the WARC record format itself.
A WARC file does not prove that every page, image, or API response was collected. A crawler can store a challenge page, truncated response, or incomplete set of resources just as faithfully as the intended content. Validate status, media type, payload quality, and crawl coverage alongside archive creation.
Replaying a JavaScript application may need resources and interactions the capture never recorded. Saving a rendered DOM or screenshot is also different from retaining the original network responses. Choose the capture method for the evidence you need, and protect archives containing cookies or private response data.
How Crawlora handles this
Structured responses from Crawlora can feed an extraction dataset. If your workflow also needs WARC preservation, plan that as an explicit capture and storage step; receiving JSON from an API is not evidence that an underlying WARC archive exists.
Related reading
FAQ
No. WARC can store HTML alongside other resource types, HTTP messages, and capture metadata. An HTML file alone does not preserve that record structure or the rest of a page's resources.
No. Replay depends on the resources captured and the replay software. Missing scripts, API responses, or referenced revisit payloads can leave a page incomplete.
Yes, when the needed response payloads were preserved and can be resolved. Track the capture record and extraction version so later results remain attributable.
Beyond WARC File (Web ARChive), Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.