How Search Engines Work: Crawling, Indexing, and Ranking in Plain English

A search engine does four things before your page can show up for anyone: it discovers the URL, crawls it, indexes it, and ranks it against everything else it knows about. Skip or fail any one stage and the page is invisible, no matter how good it is. Most "why isn't my page on Google" problems are really "my page got stuck at one of these four stages" problems, and knowing which stage tells you exactly what to fix.

This walks through all four in plain language, no jargon left undefined, using Google as the reference because it is the one most people are optimizing for. At the end, where AI answer engines like ChatGPT and Google's AI Overviews fit into this same pipeline, because in 2026 that question comes up in every SEO conversation.

Stage 1: discovery, how Google learns your URL exists

Google cannot crawl a page it has never heard of. Discovery is how a URL first enters Google's list of things to look at, and there are two main ways it happens.

The first is links. When Google crawls a page it already knows and finds a link to a new URL, it adds that URL to its crawl queue. This is the original mechanism of the web and still the dominant one: a page that nothing links to, internally or externally, is hard for Google to discover. That is why internal linking matters for more than navigation; it is how Google finds and reaches your deeper pages. We go into that in the internal linking strategy for SEO.

The second is sitemaps. An XML sitemap is a file that lists the URLs you want Google to know about, which you submit in Search Console. It does not guarantee crawling or indexing, but it is a direct way to tell Google "these URLs exist," which matters for new sites with few inbound links and for large sites where some pages are buried. Generate and check one with our XML sitemap generator. If Google does not know your URL exists, nothing downstream can happen.

Stage 2: crawling, fetching and rendering the page

Once a URL is in the queue, Googlebot, Google's crawler, fetches it, the same way a browser requests a page. But modern crawling has a second step that trips up a lot of JavaScript-heavy sites: rendering.

Many sites today build their content with JavaScript in the browser rather than sending finished HTML from the server. Google handles this by rendering the page, running the JavaScript to see the final content, but rendering happens in a separate, queued step that can lag behind the initial fetch. So content that only appears after JavaScript runs may be seen later, or, if the script fails or is blocked, not at all. The practical lesson: important content and links should be present in the HTML where possible, not dependent on client-side scripts to appear.

Crawling is also where robots.txt applies. That file tells Googlebot which paths it may fetch. Block a path and Googlebot will not crawl it, which is useful for saving crawl activity on junk URLs but a mistake if you block resources the page needs to render, like CSS and JavaScript. Check your file with our robots.txt generator, and note that blocking a page in robots.txt is not how you remove it from results; that is a different mechanism covered in robots.txt vs noindex.

Stage 3: indexing, deciding what to store

Crawling a page does not mean Google will index it. Indexing is the stage where Google processes what it crawled, decides the page is worth storing, and adds it to the searchable index. Two things about this surprise people.

First, the index is selective. Google does not index every page it crawls. Thin, duplicate, or low-value pages can be crawled and then deliberately left out, which shows up in Search Console as "Crawled, currently not indexed." That status usually means the page was seen but did not clear the quality bar to be stored, not that something is technically broken. When indexing gets stuck, our guides on how to get Google to index your site and page not indexed: Search Console fixes walk through the specific fixes.

Second, canonicalization happens here. When several URLs show the same or very similar content (a page with and without tracking parameters, an HTTP and HTTPS version, a print variant), Google picks one to represent the group, the canonical, and folds the others into it. You can signal your preference with a canonical tag, but Google makes the final call. This is why a page can be crawled yet not appear as itself in the index; Google decided a different URL is the canonical version.

Stage 4: ranking, ordering the results

Once a page is in the index, ranking is the stage that decides where it appears when someone searches. Google evaluates the indexed pages that match a query and orders them by how well each one satisfies that specific search. There is no single score; it is many signals combined, and Google keeps the exact recipe private. In plain terms, the ones that carry the most weight are:

• Relevance. Does the page actually cover what the query is about? This is matching the meaning of the query to the content, not just keyword presence.
• Quality and trust. Is the content accurate, substantial, and from a source with some credibility on the topic? Google leans on signals of experience and expertise here.
• Usability. Does the page load reasonably, work on mobile, and not bury the content under intrusive elements? Core Web Vitals feed into this; see Core Web Vitals explained.
• Links and reputation. Do other credible pages reference this one? Links still function as votes, weighted by the trust of the linking site.

The point to hold onto is that ranking only operates on pages that made it through the first three stages. Chasing ranking signals for a page that is not even indexed is effort spent on the wrong stage. Confirm the fundamentals first: is the page discoverable, crawlable, and indexed? You can sanity-check a page's basics by running it through a free SEO analyzer that reports its title, headings, and meta tags the way a crawler sees them.

Where AI answer engines fit in 2026

AI answer engines (ChatGPT search, Perplexity, Google's AI Overviews) do not replace this pipeline; they sit on top of it. Each one draws candidate sources from an underlying search index: ChatGPT search leans on Bing, AI Overviews use Google's index, Perplexity runs its own crawl plus conventional results. Then a language model summarizes those sources and cites some of them.

The consequence is that the four stages above are still the price of entry for AI visibility. If your page is not discovered, crawled, and indexed, it is not a candidate for an AI answer any more than for a blue link. What changes at the AI layer is selection and presentation: the engine favors pages that state a clean, extractable answer and often cites only a few. So the base remains classic SEO, with an added emphasis on clear, directly-answered content. We cover the specifics in how to get cited by ChatGPT, Perplexity, and Google AI Overviews.

Frequently Asked Questions

What are the stages a page goes through to rank on Google?

Four: discovery (Google learns the URL exists via links or a sitemap), crawling (Googlebot fetches and renders the page), indexing (Google decides the page is worth storing in its searchable index), and ranking (Google orders indexed pages for a given query). A page must pass all four to appear in results.

What is the difference between crawling and indexing?

Crawling is Google fetching and rendering a page. Indexing is Google deciding that page is worth storing in its searchable index. A page can be crawled but not indexed, shown in Search Console as "Crawled, currently not indexed," usually meaning it was seen but did not meet the quality bar to be stored.

Why is my page crawled but not indexed?

The index is selective; Google does not store every page it crawls. Thin, duplicate, or low-value pages are often left out. It can also be a canonicalization decision, where Google folded your URL into a similar one it chose as the canonical. Improving the page's depth and uniqueness is the usual fix.

Does Google read JavaScript content?

Yes, but in a separate rendering step that can lag behind the initial crawl. Content that only appears after JavaScript runs may be seen later, or not at all if a script fails or a needed resource is blocked. Putting important content and links in the server-sent HTML avoids the risk.

How do AI answer engines like ChatGPT fit into how search works?

They sit on top of the same pipeline. AI engines pull candidate sources from an underlying search index (Bing for ChatGPT, Google's index for AI Overviews), then a model summarizes and cites some of them. Your page still has to be discovered, crawled, and indexed to be a candidate, so classic SEO remains the foundation.