Home › Knowledge Base

What is crawlability?

What a crawler actually does

A crawler is just a program with a list. It asks your server for a page, reads the code that comes back, and follows the links it finds there. Then it does exactly the same thing on the next page in the list.

Nothing magic happens at any point. If no link points at a page, the crawler has no path to it, and if the page needs a login, the crawler stops at the door.

That is why site structure matters more than most owners expect. The links between your pages are the only map a crawler is ever handed.

None of this is complicated. It is just bookkeeping.

What usually blocks a crawler

Most crawl problems are plumbing rather than writing, and the same handful of causes come up again and again.

None of them are hard to put right once you know they are there, and most take a single change to a setting.

• A page with no link pointing at it from anywhere else on the site.
• A robots.txt rule that quietly tells crawlers to stay out of a whole folder.
• A noindex tag that somebody left in place after the site went live.
• Content that only appears once a script runs, with nothing in the code behind it.
• Slow or flaky hosting that times out on the days a crawler calls.

How to check your own site

Open one of your pages in a private browser window. If it loads without a login and its words appear in the page source, a crawler can usually load it too.

Then look at your sitemap and your robots file together. Google Search Console lists the pages Google fetched, the pages it skipped, and the reason it gives for skipping each one.

Fix the plumbing before you touch a single word of the copy, because a page nobody can reach cannot be genuinely useful to anyone.

Then look at a page you added recently, because new pages are the ones most likely to be missing a link from anywhere else.

Check this once a quarter. It takes ten minutes. Most sites pass, and the ones that fail usually fail for one reason.

How Microsite Generator handles this

Sites built here ship with a page structure and internal links already in place, so every page has a path in from the moment it publishes. Sites are capped at up to 25, 50 or 100 pages each, and each page is individually built from what you tell the agent.

The compliance gate runs before anything publishes, and every page is checked against Google Search Essentials and modern web standards. You also choose whether Google may use your content to train its AI, and search indexing is unaffected either way — the two settings are separate.

Crawlability is not something you should have to think about every week. Set the structure once, then spend your time on the words instead.

Updated September 2026