Discover HTML documents
Begin with a public URL and process HTML responses as individual documents. Each fetched document can reveal navigation, content relationships and additional crawl targets.
HTML CRAWLER
HTML Crawler for Authorized Website Capture
Inspect the HTML layer of an authorized public website to discover documents, follow same-site links and map the resources referenced by each page before creating portable output.
Follow same-site links
Links found in page markup can be normalized and added to the crawl queue when they belong to the authorized website. This creates a document-level map instead of treating the starting URL as the complete site.
Record referenced resources
HTML commonly points to stylesheets, JavaScript, images, icons and fonts. Recording these dependencies is important when the intended result must reproduce rendered pages away from the original host.
Prepare portable HTML output
For static capture, page and asset references need to remain meaningful after files are moved. Crawling provides the discovery layer used before paths are rewritten and the captured site is validated independently.
Frequently asked questions
What is the difference between an HTML crawler and a website copier?
The HTML crawler discovers documents, links and referenced resources. A website copier uses discovered material to create portable static output and validate the resulting copy.
Does the HTML crawler rebuild application source code?
No. Crawling works from publicly delivered website responses. AI-assisted reconstruction is a separate workflow for creating a fresh editable implementation.
Can the crawler discover multiple pages?
Yes. Same-site links can be followed according to crawl rules so reachable public pages can be represented as a connected website structure.