The short answer
Quick answer: A search engine does almost all of its work before you type anything. It crawls the web continuously, downloading pages by following links. It indexes them, building an inverted index that maps every word to the list of pages containing it. When you search, it does not scan the web; it looks up your words in that index, intersects the lists, and ranks the matching pages using hundreds of signals about relevance and quality. The index is split across thousands of machines that are queried in parallel, which is how results come back in a fraction of a second.
Google describes these same three stages (crawling, indexing, serving) in its guide to how Search works. The ranking details are proprietary; this article covers the well-established principles.
Stage 1: Crawling
There is no central list of web pages. A crawler (Google's is called Googlebot) discovers them:
- Start from a set of known URLs.
- Fetch each page.
- Extract its links and add new ones to a queue.
- Repeat, endlessly.
Sitemaps submitted by site owners are another source of URLs.
The crawler has to make choices:
- Politeness. It limits how fast it requests pages from any one site and obeys
robots.txt, a file where sites say which paths crawlers may visit. - Prioritisation. Important and frequently changing pages are revisited often; obscure, static ones rarely.
- Deduplication. The same content often appears at several URLs. The engine picks one canonical version.
- Rendering. Many pages build their content with JavaScript. The crawler runs pages in a headless browser to see what a user would see. See how browsers render a web page.
Stage 2: Indexing
Each fetched page is processed:
- Extract the text, title, headings, links and structured data.
- Work out language, topic and freshness.
- Break the text into tokens and normalise them (lower-casing, handling word forms).
Then comes the central data structure.
The inverted index
A normal ("forward") index says: document 1 contains these words. An inverted index flips it:
database -> [doc 3, doc 17, doc 42, doc 108, ...]
index -> [doc 3, doc 9, doc 42, doc 77, ...]
tutorial -> [doc 9, doc 42, doc 300, ...]
Each list is called a posting list. It stores document IDs in sorted order, along with word positions and other data.
To answer database index tutorial:
- Fetch the three posting lists.
- Intersect them. Because they are sorted, this is a fast merge. Document 42 is in all three.
- Score the surviving documents.
This is the same idea as an index at the back of a book, applied to the whole web. It is closely related to database indexes, and it is what search engines like Elasticsearch use too.
Posting lists are heavily compressed, and rare words have short lists, so the engine starts with the rarest term to keep the work small.
Stage 3: Ranking
Millions of pages may match. The value of a search engine is in ordering them.
Relevance to the query
- Term frequency. A page that mentions the query terms prominently is more likely to be about them.
- Rarity. A match on a rare word says more than a match on a common one. Classic scoring formulas such as TF-IDF and BM25 capture both.
- Placement. Matches in the title, headings and URL count more than matches deep in the text.
- Proximity. Query words appearing close together, or as an exact phrase, score higher.
Authority: PageRank
In 1998, Larry Page and Sergey Brin described the idea that made Google stand out in The Anatomy of a Large-Scale Hypertextual Web Search Engine. PageRank treats a link from page A to page B as a vote for B. Votes from important pages count more, and importance is computed recursively over the whole web's link graph.
The insight was that the web's own structure reveals which pages people consider worth pointing to. Links remain a signal today, among many others.
Understanding meaning
Keyword matching fails when the query and the page use different words. Modern engines use machine learning to bridge the gap:
- Query understanding. Spelling correction, synonyms, recognising names and places, and working out intent. "Apple price" could be about fruit or shares.
- Semantic matching. Language models represent queries and passages as vectors so that text with similar meaning matches even without shared words. See how embeddings work and transformers explained.
Other signals
- Freshness, for queries about recent events.
- Location and language of the searcher.
- Page experience: mobile friendliness, loading speed, security.
- Quality and spam detection: demoting pages that try to manipulate rankings.
Ranking in stages
Running the most sophisticated models on millions of candidates would be too slow. Ranking is a funnel: a cheap scoring pass selects the top few thousand candidates, and progressively more expensive models re-rank smaller sets. The same pattern appears in recommendation systems.
How it answers in milliseconds
- Sharding. The index is far too big for one machine. Documents are divided among many shards. See sharding explained.
- Scatter and gather. A query is sent to all shards in parallel. Each returns its best matches, and a coordinator merges them into a final top list.
- Replication. Each shard has many copies, for capacity and to survive failures.
- Tail tolerance. With thousands of machines involved, one is always slow. The system sends backup requests or returns without the slowest shard rather than waiting.
- Caching. Popular queries are answered from cache. A small fraction of queries makes up a large fraction of traffic.
- Early termination. The engine stops scoring once it is confident it has the top results.
- Memory. Hot parts of the index are held in RAM.
- Data centres everywhere. Your query is served from one near you.
The results page
Once the top documents are chosen, the engine builds the page: titles, snippets showing your query words in context, and special features such as direct answers, maps, images, knowledge panels and, increasingly, AI-generated summaries. Ads are selected by a separate auction system that runs alongside.
What this means for site owners
Search engine optimisation follows directly from the pipeline:
- Crawlable: pages must be reachable by links and not blocked.
- Indexable: content should be in the rendered HTML, with one canonical URL.
- Relevant: clear titles, headings and text that answer real queries.
- Trustworthy: good content earns links.
- Fast and usable, especially on mobile.
Frequently asked questions
What is an inverted index?
A data structure that maps each word to the list of documents containing it, so that finding all pages with a given word is a direct lookup.
What is PageRank?
An algorithm that scores a page's importance from the number and importance of the pages linking to it.
Does Google search the live web when I type a query?
No. It searches its own index, built in advance from crawled pages. That is why a new page takes time to appear and a deleted one can linger.
How does a new page get into search results?
A crawler discovers it through a link or a sitemap, fetches and renders it, and adds it to the index. This can take from hours to weeks.
Conclusion
A search engine is fast because the hard work happens ahead of time. Crawling gathers the web, the inverted index turns "find these words" into a lookup, and layered ranking narrows millions of matches to ten. Splitting the index over thousands of machines and querying them in parallel does the rest.
Related articles
- How a Database Index Makes Queries 1000x Faster
- How Embeddings Turn Words Into Numbers
- Sharding Explained: How Databases Scale Beyond One Machine
- How Browsers Render a Web Page
Sources and further reading
