What Is robots.txt?
robots.txt is a plain text file at the root of your website that tells search engine crawlers which pages they are allowed or not allowed to access. It is part of the Robots Exclusion Protocol.
Important: robots.txt controls crawler access, not indexing. Blocking a URL in robots.txt prevents crawlers from visiting it but does not prevent the URL from being indexed if it has inbound links.
Basic Syntax
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /private/
Sitemap: https://yourdomain.com/sitemap.xmlWhat to Block
- Admin and authentication pages
- API endpoints
- Search result pages (prevents duplicate content crawling)
- User account pages
- Staging or development paths
- Checkout and cart pages
Common Mistakes
- Blocking your own sitemap — always reference your sitemap in robots.txt
- Accidentally blocking everything —
Disallow: /blocks all crawling - Blocking CSS and JS — Googlebot needs these to render pages correctly
- Using it to hide pages — blocked pages can still appear in search if linked from elsewhere
Crawl Budget
By blocking low-value pages (search results, filter pages, session parameters), you ensure Googlebot spends its time on your important content. Critical for large e-commerce sites where faceted navigation can generate millions of URL combinations. Audit crawl waste with our index checker or SEO indexing software dashboard.