Cost of retrieval: why fewer, clearer pages can outrank more

What cost of retrieval means for a search engine and your site

Cost of retrieval is what a search engine spends to crawl, understand, store, rank and serve your pages, weighed against what those pages give back. A site that is cheap to process for the value it offers gets more of its pages crawled, indexed and shown. A site full of pages that cost effort and return nothing gets less.

Think of a warehouse that stores your stock for free, as long as people buy it. Every box you send costs shelf space and handling. Send ten boxes of the same thing and nine sit unsold. Eventually the warehouse stops taking your deliveries as fast.

Koray Tuğberk Gübür brought the term into SEO. He is the SEO who put topical maps and topical authority into the industry's vocabulary (his whole framework is in Koray Tuğberk Gübür's topical map framework). In a 2022 interview with Reboot Online he defined it: "Cost of retrieval is the ratio of the cost experienced during crawling, understanding, evaluating, indexing, ranking, and serving, and the value obtained afterwards." In the same interview he says the concept comes from Trystan Upstill, whom he names as his source for it.

His rule of thumb, from a 2026 Search Engine Land article, is one sentence: "The cost of ranking a site can't exceed the cost of not ranking a site."

Where it sits in his topical authority formula, by date

Koray has published his topical authority formula in three versions:

• 2021: "Topical Authority = Topical Coverage * Historical Data" (Oncrawl case study). No cost term yet.
• 2026: historical data times topical coverage, "divided by 'cost of retrieval'", then "interpreted with visual semantics" (Search Engine Land, local SEO).
• 2026, later: "((Historical data x Topical coverage) ÷ Cost of retrieval) x Right visual annotations" (Search Engine Land, visual semantics).

Dividing is the point. However complete your coverage and however long your history, a high cost shrinks the result. In the local SEO article he spells out the practical meaning: "every unnecessary page increases the cost of retrieval."

What search engines document about the cost of your pages

Google documents that crawling is finite, and Google, Bing and Yandex all say duplicate or low-value URLs waste it. Google's court record adds the scale: it crawls trillions of pages, and de-duplication cuts the links it extracts by about nine in ten before indexing starts.

Google's own documentation

• Crawling is limited. Google's crawl budget documentation says its resources "are finite, and we must prioritize resource allocation across the web."
• Crawled does not mean indexed. The same page: "not every page that is crawled will necessarily be indexed. After crawling, each page must be evaluated, consolidated, and assessed to determine its suitability for the index."
• Your URL count is the lever you hold. Google calls it perceived inventory. If many known URLs are duplicates or unimportant, "this wastes a lot of Google crawling time on your site. This is the factor that you can positively control the most."
• Low-value URLs drain the good ones. A 2017 Search Central post ranks six kinds of low-value URL by how much they hurt: faceted navigation and session IDs, on-site duplicate content, soft error pages, hacked pages, infinite spaces and proxies, and low quality or spam content. "Wasting server resources on pages like these will drain crawl activity from pages that do actually have value."
• Rendering costs too. In a December 2024 post, Google wrote that crawling the files a page needs to render "will chip away from the crawl budget" of the host serving them.
• There is a size cut-off. Googlebot's documentation says it crawls the first 2MB of an HTML file for Google Search and stops there.

Bing, Yandex and IndexNow say the same

Microsoft is the most direct. Section 6 of the Bing Webmaster Guidelines reads: "Duplicate URLs dilute signals and reduce Bing's confidence in selecting a URL for grounding results or citations." Section 21 says Bing allocates crawl capacity by "site health, efficiency, signal quality, and crawl value", and that crawl waste, duplication or low-value URLs "can directly limit indexing". Those rules also cover Copilot's answers, not only Bing's results.

Yandex's help page on duplicate pages says crawling them "wastes time as well as the resources of your site and Yandex Search", may delay new pages and may make duplicates compete. The IndexNow FAQ, the shared submission protocol Bing and Yandex use, adds a detail most people miss: every URL you submit "counts toward your site's crawl quota."

What the Google trial showed about index size and cost

The US antitrust case against Google put numbers on the record. The remedies opinion of 2 September 2025 (pages 137 to 138) says "Google starts by crawling trillions of web pages", that "lots of pages ... are filled with spam or are duplicates", and that Google scores what it crawls and works to exclude pages without value from its index. It adds that quality and popularity signals decide how often Google crawls a page.

Pages 144 to 145 quote a Google exhibit: de-duplication lets Google "reduce the 1 trillion web links" it extracts to the "100 [billion]" it processes for possible indexing. That is a cut of about nine in ten before indexing even starts. The 2024 liability opinion (finding 52) gives the bill: "In 2020, Google spent $8.4 billion to operate its search engine".

Search engine research says the same, in older words

Information retrieval research has treated cost as a design problem for decades. Wang, Lin and Metzler's cascade ranking model (SIGIR 2011, the main information retrieval research conference) runs cheap ranking functions first. Expensive ones run on a shrinking set, in a cascade that "progressively prunes and refines the set of candidate documents". Carmel and colleagues' static index pruning (SIGIR 2001) removed index entries that barely affect relevance "and yet have only a minor effect on retrieval results". In both, a document that adds little is the first thing an engine learns to skip.

Where the documented part ends and Koray's interpretation begins

Search engines document the inputs: crawling is finite, duplicates are wasted, low-value URLs are filtered, and a site rarely gets more than two listings for one search. Koray's step is to treat all of it as one cost that divides topical authority. That step is his model, not an engine statement.

Google has spoken to the strongest form of the idea. When CNET deleted thousands of old articles in 2023, Danny Sullivan, Google's public liaison for Search, told Gizmodo that removing a weak page "might mean, if you have a massive site, that we're better able to crawl other content on the site. But it doesn't mean we go, 'Oh, now the whole site is so much better'".

What raises the cost of retrieval on a real site

Five things raise the cost of retrieval on a real site: duplicate and near-duplicate pages, thin pages, pages off the site's subject, slow or broken pages, and a structure that hides pages or multiplies URLs.

A duplicate page can take its links with it

The cost of a duplicate is not only the wasted crawl. Google's own research paper on the problem, Detecting Near-Duplicates for Web Crawling (Manku, Jain and Das Sarma, WWW 2007), deals with a newly crawled page that nearly copies one already crawled. It says the crawl engine "should ignore" the duplicate "and all its out-going links". It adds that dropping near-duplicates "saves network bandwidth, reduces storage costs and improves the quality of search indexes."

Put that next to a site's internal links. If the only link to your crown and filling guide sits on a city page that copies the service page, that link may be thrown away with the copy. The paper assumes such links lead to pages the crawler can already reach from the page it keeps. A page linked only from a copy is the case that assumption misses. That risk is my reading, not the paper's claim. A link plan only works on pages the engine keeps. The paper is from 2007 and describes a design, not today's exact crawler. Treat it as a strong reason to put links on the page you keep, not as a measured rule.

Duplicates also split the signals you do earn. Two pages chasing one search share the links and clicks that one page could have had, which is keyword cannibalization. And links between pages only help when they connect pages that are both kept, as internal links inside a topical map explains.

What lowers it, and two fixes that do not work

Merging pages that answer the same need lowers the cost most. After that come removing pages with no purpose, keeping one URL per page, a fast and stable server, and links to your strongest pages from the homepage.

Two fixes that do not save what people think

Noindex does not save the crawl. Google's crawl budget page says "Don't use noindex" to manage crawling, because Google "will still request, but then drop the page", which is "wasting crawling time". Noindex keeps a page out of results; it does not make it cheap.

Blocking pages does not hand their budget to other pages. The same page: Google "won't shift this newly available crawl budget to other pages unless Google is already hitting your site's crawl capacity limit." On most small sites, blocking only stops the crawl of what you block.

Why every page needs a purpose, and a query is not always a page

A page with no purpose still costs the engine a crawl and a judgment, and returns nothing for it. That is the reasoning behind two rules we hold on every site we work on: every page must earn its place, and many searches belong in a section, not on a page of their own.

The first rule is about pages already live. Any website that has worked with a bad agency before has a large share of off-topic posts, because that agency "just worked to publish as much as possible ... because that's what was in the retainer, but that's not how SEO is done properly. It's about making sure the website is a quality website and every single page has a purpose on that website. If a page does not have a purpose, it does not deserve to be on the website." Before anything goes, I check whether it can be merged into a page that is more relevant.

The second rule is about pages not yet written. One of the mistakes we see in maps buyers build themselves is a page for every search. "A query does not deserve its own page" when it is really a section of a page that already exists. Each extra page is one more URL to crawl, one more candidate for the same search, and one more place for links to scatter.

Koray formalises the second rule as Query Deserves a Page. In his 2026 local SEO article, whether a search earns its own page depends on four checks: "The query has high search demand", "The query has different entities", "The query has low similarity" and "The query has a pattern". If they do not hold, it becomes a section, a table or a card on an existing page. His list of how his team lowers the cost starts with "Removing unnecessary pages through the Query Deserves a Page principle."

We read the search results for both versions of a search before deciding, and that test, with ten real pairs, is in when a search deserves its own page.

A made-up before and after: one site section, rebuilt

Here is a made-up example, invented for this lesson: a dental clinic in Plano with one office, and its teeth whitening section. The section grew to 15 URLs over the years. After a purpose check it is 5 pages, and every real question a patient asks still has an answer.

Every merged URL gets a permanent (301) redirect to the page that took its content, so visitors and links arrive in the right place.

Count what changed for the engine, using the same made-up numbers:

• URLs to crawl and judge in the section: 15 before, 5 after (plus redirects until they fade).
• Pages competing for "teeth whitening" near the clinic: 4 before (the service page and three city copies), 1 after. Google's site diversity system generally shows two at most anyway.
• Pages competing for the price search: 2 before, 0 separate pages after. The price is a section of the page that sells the treatment.
• Pages off the subject: 1 before, 0 after.

How to estimate your site's cost of retrieval by hand in Search Console

No tool shows a cost of retrieval number, and Koray says none can measure topical authority either. You can still estimate how much of your site costs the engine effort for nothing. It takes three Search Console reports and an afternoon.

1. Open Indexing, then Pages. Write down the Indexed total and the Not indexed total.
2. Copy the not-indexed reasons that mean effort without value. Google's Page indexing report help defines them. "Crawled - currently not indexed": crawled, judged, not kept. "Duplicate without user-selected canonical" and "Duplicate, Google chose different canonical than user": a copy Google will not serve. "Soft 404": a page that says "not found" with a 200 (OK) code. Leave out "Alternate page with proper canonical", which is working as intended.
3. Work out the waste share. Add those reasons together and divide by the indexed pages plus those reasons.
4. Open Performance, then the Pages tab, for the last 3 months. Count indexed pages with no impressions at all. Those were kept but never shown.
5. Open Settings, then Crawl stats. In the Crawl stats report, note the share of requests that did not return 200 (OK), the split between discovery and refresh, and the average response time.
6. List the URLs behind each number and run the purpose check on each: merge, improve or remove.

A worked example on made-up numbers

Take a made-up site with 1,240 known URLs: 610 indexed and 630 not indexed. Among the not indexed, 260 are crawled but not indexed, 140 are duplicates without a chosen canonical, 45 are duplicates where Google picked another canonical, and 25 are soft 404s. The rest are redirects, 404s and 20 discovered but not yet crawled.

• Waste share: 260 + 140 + 45 + 25 = 470. Divided by 610 + 470 = 1,080, that is 43.5%. So 43.5% of the pages Google judged ended up crawled but not kept, or as copies it would not serve.
• Never shown: 180 of the 610 indexed pages had no impressions in 3 months, which is 29.5%.
• Crawl waste: 29% of crawl requests got a redirect, a 404 or an error instead of a 200.

These are proxies, not Google's own number. Google says plainly: "Don't expect every URL on your site to be indexed." The use of the three shares is to see where the waste sits, and whether it shrinks after you act. I write them down with the date and look again three months later.

Download the hand estimate worksheetCSV for Google Sheets or Excel. Nine rows: where each number lives in Search Console, what to copy, the made-up example, an empty column for yours, and what it tells you.

What the pruning evidence shows, and the limits of the idea

The evidence for cutting pages is mixed. I found no published controlled test showing that deleting pages lifts a whole site. The one split test on thin pages found the opposite of deleting worked better.

• Enriching beat deleting. SearchPilot, a company that sells SEO split testing, published a 2025 test on thin nutrition pages of a large retailer. The team had leaned toward deleting and redirecting them. Instead the test pages gained a specification table and related products. Over about six weeks, on one site, the test showed a nearly 20% organic traffic uplift against the control.
• Removing can go either way. In SearchPilot's 2025 review of tests, removing an expert video carousel from listing pages gained 4.1%, and removing a map from location pages lost 7%. Those removed parts of pages, not pages, but they show cutting is a test, not a rule.
• The famous case published no results. CNET's 2023 deletion of thousands of articles came with a memo promising a signal to Google, and with no before and after data. Google's answer was that deleting content only because it is old is "not a thing".
• Koray's own cleanups are uncontrolled. His 2023 Complaintsboard case paired pruning and merging with new templated content for the top 500 companies, so the share due to fewer pages cannot be separated.

Three more limits are worth keeping in mind:

• Crawl budget is a big-site problem. Google writes its crawl budget guide for sites with over 1 million pages that change weekly, or over 10,000 that change daily. On a 200-page site, the cost that matters is duplicates competing and pages judged not worth keeping, not crawl capacity.
• Nobody can compute the formula. In February 2024 Koray wrote: "Refrain from believing that anyone can measure TA [topical authority] with a tool." I read that as covering its divisor too.
• Cheap does not mean wanted. A page that costs nothing to process and answers nobody is still worth nothing. Lowering the cost helps the pages that already deserve to rank.

So the rule I work by is merge first, improve second, remove last, and measure before and after. The steps, and how to handle redirects and links, are in pruning, merging and updating. Reading the results afterwards is covered in how to tell if a topical map is working.

How cost of retrieval connects to the rest of a topical map

Cost of retrieval is the reason behind three other decisions in a topical map: how authority builds, when a search gets a page, and what happens to pages that do not belong.

• Topical authority is coverage built up over time. Cost is the divisor that keeps "more pages" from meaning "more authority".
• When a search deserves its own page is the decision that keeps cost low before a page exists.
• Pruning, merging and updating is the clean-up for pages that already exist.
• Core section and outer section decide which pages carry the subject, so you know which ones the links and the homepage should point to.

A map built this way plans fewer, clearer pages from the start, each with a job. If you would rather have that plan built for your site, that is what the growth plan is.