How the web gets found and filed

People say a search engine "searches the web". It does not, not when you type a query. By then the work is long done. A search engine does three separate jobs, and only the last happens at query time.

1. Crawling. Discovering that a page exists and fetching its content. Continuous, in the background, across the whole web.
2. Indexing. Making sense of a fetched page and storing it in a giant data structure built for instant lookup.
3. Ranking. When you search, consulting that stored index and ordering the results. Milliseconds.

The searchable web is not the live web. It is a snapshot, the index, assembled ahead of time. When you search, you are querying that snapshot, not the internet.

This lesson is about the first two jobs, the invisible pipeline that decides what is even eligible to appear. A page that is not crawled and indexed cannot rank, no matter how good it is, which makes this the part of search most people never think about and most sites get wrong.

A crawler, Google's is Googlebot, is a program that fetches a page, reads the links on it, adds those links to a queue, and repeats. The web is a graph, and the crawler walks it link by link.

This has an immediate consequence that governs a lot of SEO: a page with no links pointing to it is nearly invisible. If nothing references a page, the crawler has no path to discover it. Links are not only a ranking signal, they are the discovery mechanism.

Crawlers get help finding things:

• A sitemap, a file listing a site's URLs, hands the crawler a map instead of making it find every page by wandering.
• Internal links spread discovery through a site, which is why an orphaned page, linked from nowhere, tends to go unfound.
• External links from other sites are how a brand-new site gets discovered at all.

So the first practical question for any page is not "is it good" but "can the crawler reach it", and the honest answer for large parts of most sites is: not easily.

The web is effectively infinite and a crawler's resources are not, so a search engine spends a limited crawl budget on each site: roughly how many pages it will fetch, and how often, before moving on.

The engine allocates that budget by a rough sense of a site's importance and how quickly it responds. A large, authoritative, fast site gets crawled deeply and often. A slow site, or one that wastes the budget, gets crawled shallowly.

And budget is easy to waste. Sites generate near-infinite low-value URLs, endless filter combinations on a shop, calendar pages stretching forever, session IDs creating a fresh URL each visit. A crawler pouring its budget into millions of useless variants never reaches the pages that matter.

This is why technical SEO exists as a discipline. Much of it is not about content at all. It is about not wasting the crawler's time: blocking junk URLs, keeping the site fast, and pointing the budget at pages worth indexing. For a small site this is a non-issue; for a large one it is the whole game.

Fetching a page is not understanding it. Indexing is the step that turns raw HTML into something searchable, and the core structure it builds is worth knowing because it explains why search is instant.

The key idea is an inverted index. A normal index maps a document to its words. An inverted index flips that: it maps each word to the list of documents that contain it.

normal:    page 1 -> [running, shoes, trail]
inverted:  "running" -> [page 1, page 7, page 40, ...]
           "trail"   -> [page 1, page 12, ...]

Now a search for running trail does not scan the web. It looks up two short lists and intersects them, which is why results come back in milliseconds across billions of pages. It is the same structure a library card catalogue uses, at planetary scale.

Alongside the words, the index stores signals it will need later: the title, the page's language, when it was last updated, structured-data annotations, and a great many quality and link signals. Indexing is where a page is turned into features a ranking system can compare.

Three distinct stages, three ways to fall out, and confusing them is the most common diagnostic mistake in SEO.

Each stage is a filter, and a page can pass one and fail the next. A page can be crawled and then not indexed, because the engine judged it too thin, too similar to another page, or otherwise not worth a slot. And a page can be indexed and simply rank poorly.

The practical value is in diagnosis. "My page is not on Google" has three completely different causes with three different fixes. Search Console distinguishes them, telling you whether a page was discovered, crawled, and indexed. Treating a crawling problem as a content problem, or the reverse, wastes enormous effort, and most SEO frustration comes from working the wrong stage.

Crawling assumes the content is in the HTML the server returns. Modern sites frequently break that assumption: they send a nearly empty HTML shell and build the real content in the browser with JavaScript. A human sees a full page; the crawler's first fetch sees almost nothing.

Search engines handle this with a second, heavier pass called rendering: they run the JavaScript, as a browser would, to see the final content. But rendering is far more expensive than reading HTML, so it is often deferred, done later, sometimes much later, and not guaranteed for every page.

The consequences are practical and common:

• Content that only appears after JavaScript runs may be indexed slowly, or on a large site, missed.
• If a page depends on a click or scroll to load its main content, a crawler that does neither never sees it.

The robust answer is server-side rendering: send the finished content in the initial HTML so the crawler gets it on the first, cheap pass. The general principle is blunt and worth remembering: if the crawler cannot see your content without running your code, assume it might not run your code.

Sites are not passive in this. A few standard controls let a site steer crawling and indexing, and confusing two of them causes a classic, damaging mistake.

• robots.txt controls crawling: it asks crawlers not to fetch certain paths. It is a request, honoured by mainstream engines, not a lock.
• A noindex tag controls indexing: it lets a page be fetched but asks that it not be stored.
• A canonical tag handles duplicates: it tells the engine which version of near-identical pages is the real one to index, consolidating signals onto it.

The classic error follows directly from the crawled-versus-indexed distinction. To keep a page out of search you use noindex, and the engine must crawl the page to see that tag. If you also block the page in robots.txt, the crawler never fetches it, never sees the noindex, and the page can linger in results. The two tools operate at different stages, and using the crawl control to enforce an indexing decision quietly fails.

The lesson generalises: know which stage a control acts on, because a fix aimed at the wrong stage does nothing, or the opposite of what you intended.

Discovery through links and sitemaps, a crawl bounded by budget, rendering for JavaScript, then indexing into an inverted structure. Only what survives all of this is eligible to rank.