What is a Crawler?

A bot that follows links to read pages, the way search engines discover and index websites.

Googlebot and Bingbot crawl your site to decide what to show in results. A noindex tag, a robots.txt rule or an accidental password gate can stop them, and your pages quietly drop out of search.

This service's scanner crawls up to ten pages of your site, politely and identified, to see what it runs.

What to do

Check Search Console for crawl errors monthly.
Keep robots.txt short; most sites need none.
A staging site left crawlable competes with the real one.