robots.txt
What is robots.txt?
robots.txt is a plain-text file at the root of a domain that tells crawlers which paths they may or may not request.
Why it matters
It controls crawling, not indexing: a blocked URL can still appear in results if other sites link to it.
Use noindex meta tags, not robots.txt, when you want a page kept out of the index.
Related terms
- XML sitemap
An XML sitemap is a machine-readable file listing the URLs you want search engines to crawl, along with optional metadat…
- Crawl budget
Crawl budget is the number of pages a search engine is willing to crawl on your site in a given period. It matters mainl…
- Indexing
Indexing is the step where a search engine stores and organises a crawled page so it can be returned in results. A page …
Cite this definition
Guest Post Website. “robots.txt.” Guest Posting & SEO Glossary. https://guestpostwebsite.com/glossary/robots-txt