Crawl budget
What is crawl budget?
Crawl budget is the number of pages a search engine is willing to crawl on your site in a given period. It matters mainly for large sites with thousands of URLs.
Why it matters
Waste comes from duplicate URLs, faceted navigation, infinite parameters, and soft 404s.
Small sites rarely have a crawl budget problem; they have a demand problem.
Related terms
- Indexing
Indexing is the step where a search engine stores and organises a crawled page so it can be returned in results. A page …
- Canonical tag
A canonical tag tells search engines which URL is the preferred version of a page when several URLs show similar content…
- robots.txt
robots.txt is a plain-text file at the root of a domain that tells crawlers which paths they may or may not request.…
Cite this definition
Guest Post Website. “Crawl budget.” Guest Posting & SEO Glossary. https://guestpostwebsite.com/glossary/crawl-budget