Google does not crawl your entire site every day. It visits with a limited time budget, follows some links, skips others, and leaves.
What it finds during that visit determines what gets indexed, what gets refreshed, and what stays invisible.
Your internal links are the roadmap for that visit.
If the map sends Google down dead ends, through redirect chains, or into pages nobody searches for, your most important content may never get crawled at all.
For small sites (under a few thousand pages), this rarely matters. For anything above 10,000 pages, crawl budget becomes the bottleneck between content and rankings.
What crawl budget actually is
Crawl budget is the number of pages Google can and wants to crawl on your site in a given period. It has two components, and your internal links directly control one of them.
Think of it like a building inspector with a one-hour visit. They can only check so many rooms.
They start at the front door and follow the hallways. Rooms connected to the main corridor get inspected. Rooms behind three locked doors at the end of a basement hallway do not.
If most hallways lead to storage closets, the inspector runs out of time before reaching the offices that matter.
| Component | What it means | What controls it |
|---|---|---|
| Crawl capacity | How many pages Google can crawl per visit | Server speed, reliability, error rate |
| Crawl demand | How many pages Google wants to crawl | URL popularity (links), content freshness |
Crawl demand is where internal links come in. Google's documentation states it directly: URLs that are more popular get crawled more often. Pages that nobody links to get crawled less, or not at all.
This connects directly to how backlinks and internal links work as one system: backlinks bring authority in, internal links tell Google where to spend its crawl budget.
If your site has fewer than a few thousand pages, crawl budget is almost never a problem. Google says this explicitly.
Past 10,000 URLs, or with faceted navigation and parameter URLs, Google cannot visit everything. Your linking structure determines what it prioritizes.
How internal links control what Google crawls
Every internal link is a vote for "crawl this next." Pages with many links from important pages get crawled frequently. Pages with zero links may never be crawled at all, even if they appear in your sitemap.
Three mechanisms:
Links set the crawl path. Googlebot follows links from page to page. A page one click from the homepage gets crawled almost every visit. A page five clicks deep may wait weeks.
- ✓ Linked from homepage → crawled daily
- ✓ Linked from a well-crawled category page → crawled weekly
- ❌ Linked only from a deep archive post → crawled monthly, maybe less
This is why flattening your architecture through internal linking matters. Fewer clicks from homepage = more frequent crawls.
Links signal importance. Google uses link popularity as a direct input to crawl demand. The more internal links a page receives, the more Google considers it worth recrawling.
- ✓ Product page linked from 10 category pages + 3 blog posts → "this matters"
- ❌ Product page with 1 link from a paginated archive → "this can wait"
Fresher crawls mean fresher index entries. For time-sensitive content, that translates directly to better rankings.
Bad links waste the budget. Every link to a redirect, a 404, or a low-value page is a request Google spends on the wrong thing.
Botify's research found that orphan pages consume 26% of crawl budget on average. In extreme cases, 70%.
Here is what that looks like in practice. This Unveil SEO Link Graph shows a real site with roughly 2,900 pages.
Blue nodes = pages connected to the structure. Black dots = orphan pages: 2,065 out of 2,800, nearly 75% of the site.
Unveil SEO Link Graph showing 2,065 orphan pages out of 2,800 total
Every black dot is a page Google may try to crawl through the sitemap or old backlinks. No internal links, no structural context, no chance of ranking.
What to do about it
You cannot increase your crawl budget directly. But you can stop wasting it. Three actions that make Google spend its visits on the pages that matter.
1. Link important pages closer to the surface.
Any page more than three clicks from the homepage risks infrequent crawling. Unveil SEO's depth analysis shows which pages are buried.
A single contextual link from a well-crawled page can move a target from "crawled monthly" to "crawled weekly."
2. Stop linking to dead ends.
Every link to a redirect, a 404, or a blocked page is wasted budget. Audit your internal links regularly:
- Fix broken links (404s)
- Update redirect targets to final URLs
- Remove links to pages blocked by robots.txt
3. Clean your sitemap.
If your sitemap includes URLs outside your site structure (expired products, old filters, parameter variants), Google will crawl them anyway.
Keep it lean: only URLs you want indexed, all reachable through internal links.
