How Google crawls, indexes and ranks pages

When people say a page "is not ranking", they usually mean one of three quite different things. The page might not be known to Google at all. It might be known but not stored in the index. Or it might be indexed but not chosen for the queries they care about. Google's guide to how Search works splits the process into exactly these stages, and learning to tell them apart is the single most useful diagnostic skill in SEO.

Stage 1: crawling

Google has no central registry of web pages. It discovers URLs mostly by following links from pages it already knows, and by reading sitemaps that site owners submit. The program that does the fetching is called Googlebot, and Google's documentation notes that most crawling today uses Googlebot Smartphone, which sees your site the way a mobile browser does.

Discovery and crawling are separate steps. Google can know a URL exists without having fetched it yet, which is what the Search Console status "Discovered, currently not indexed" means. Google decides how much and how often to crawl based on how important pages seem and how quickly your server responds. A slow or error-prone server makes Googlebot back off.

During crawling, Google also renders the page: it runs the JavaScript, much as a browser would, using a recent version of Chromium. Content that only appears after JavaScript runs can be seen, but only after rendering succeeds, so a blocked script file or a failing API call can leave Google looking at an empty page.

Things that commonly stop crawling:

• A Disallow rule in robots.txt that covers the URL.
• Server errors (5xx) or timeouts when Googlebot requests the page.
• The URL being reachable only through a form, a search box or a button that runs JavaScript, with no normal link pointing to it.

Stage 2: indexing

Once a page is fetched, Google tries to understand it: the text, the title, headings, images, links and structured data. It also groups pages with the same or very similar content and picks one URL to represent the group. That chosen URL is the canonical, and only the canonical is usually shown in results.

Indexing is not guaranteed. Google's documentation is explicit that it does not index every page it crawls. Pages can be left out because they carry a noindex directive, because they duplicate another page, or because Google judged them low in value. In Search Console this shows up as "Crawled, currently not indexed": the page was fetched, and Google chose not to store it for now.

Stage 3: serving and ranking

When someone searches, Google looks through its index for pages that match the query and orders them using many signals: what the query means, how relevant and useful each page is, the searcher's location and language, the device, and the type of results that serve the query best. Google describes this as a set of ranking systems working together, rather than a single score.

This is the stage most people mean by "SEO", yet it only matters for pages that survived the first two stages.

Diagnosing which stage has the problem

Search Console gives you the tools to place a problem in the right stage. The URL Inspection tool reports whether a specific URL is on Google, when it was last crawled, which canonical Google selected and whether indexing was allowed. The Page indexing report groups every known URL by status, so you can see patterns across a site.

Why this matters for everything else

Almost every technique in SEO acts on one of these stages. Sitemaps and internal links serve discovery. Robots rules, server performance and status codes govern crawling. Canonicals, noindex, structured data and content quality act on indexing. Relevance, intent and page experience shape ranking. When you learn a new technique, ask which stage it affects. That question will stop you from applying a ranking fix to a crawling problem.