Website crawler

Enter a starting URL. The crawler follows internal links from there.

What a crawl reveals that a single-page check cannot

Many problems arise between pages: repeated descriptions, links to deleted offers, and pages with no internal links. You can only find these by checking the website as a whole.

The crawler starts at your URL, follows internal links and checks up to one hundred pages. It stays on the same domain. Each page is fetched once, with a pause between requests to minimise server load.

What the report highlights

• Error pages and redirect chains — a 404 in the menu loses visitors; a chain of three redirects adds latency and weakens linking.
• Duplicate titles and descriptions — a very common finding. Google then chooses which page to show, and may not choose the one you intended.
• Thin pages — pages with only a few lines of text and little independent value. Merging them is often better than optimising them separately.
• Orphan pages — sitemap URLs with no internal links. They exist, but visitors and search engines may struggle to find them.
• Non-indexable pages — noindex directives, canonicals pointing elsewhere or robots restrictions. Sometimes intentional, sometimes forgotten development settings.

What to fix first

Start with error pages, then indexability. Next address duplicate titles and descriptions, which influence search result clicks. Leave thin content until last: improving text takes more time.

Download the report as CSV to work through it and check again later. Results are not stored here, and crawl result pages are excluded from indexing.