.png)
When it comes to search engine optimisation, most businesses are focused on ranking pages and getting traffic. But what’s often overlooked is how search engines crawl your site in the first place. That’s where the humble robots.txt file comes in.
It’s a small text file with a big responsibility — directing search engine bots on what to crawl and what to skip. When used properly, it can save crawl budget, tidy up how your site appears in search results, and stop unnecessary load on your server. Used incorrectly, it can quietly block your entire site from search engines.
Let’s break down what this file does, why it matters for SEO, and how to use it wisely.
The robots.txt file sits in the root directory of your website (e.g. https://www.example.com/robots.txt) and acts like a signpost for search engine crawlers. It tells them which areas of your site they shouldn’t crawl.
But here’s the catch: robots.txt only controls crawling, not indexing. That means a page blocked by robots.txt can still appear in search results — without any content — if it’s been linked to from elsewhere.
Also, not all bots respect robots.txt rules. Major ones like Googlebot do, but malicious crawlers usually don’t.
Here are the most common commands used in a robots.txt file:
Important: A Disallow does not stop a page from being indexed. If the page is linked to from elsewhere, it can still appear in search results. To remove it from indexing, you need a noindex meta tag on the page (and that only works if the page is crawled!).
Not every site has a robots.txt, and for small websites, it’s often not critical. But for larger sites — especially in e-commerce, finance, real estate or publishing — it becomes essential for managing crawl behaviour.
Here are four key things to consider:
You can check by visiting yourdomain.com/robots.txt. If it returns a 404, then there is no file. That’s fine by default — search engines will try to crawl everything — but you’re missing the opportunity to control that process.
While User-agent: * is common and applies to all bots, some advanced strategies involve targeting specific ones like Googlebot, Bingbot, or even niche crawlers used by SEO tools.
This can help if you want to treat bots differently — for example, allow Google full access but restrict a third-party crawler.
The format must be precise. A misplaced slash or incorrect directive can cause major SEO issues. For example:
Disallow: / → blocks the entire site.
Disallow: /shop/ → blocks only the /shop/ section.
Allow: /shop/product-page → allows this specific page even if /shop/ is disallowed.
Also check that your robots.txt file returns a status code of 200. A misconfigured server might accidentally return a 403 (forbidden) or 404 (not found), which means search engines will ignore your rules entirely.
Sites with faceted navigation (like /shoes?colour=red&size=10) can end up wasting crawl budget on near-identical pages. You can use robots.txt to block certain patterns or directories — but be cautious. Over-blocking can lead to poor indexing of important content.
Robots.txt is useful in a variety of scenarios. Some of the most common include:
The robots.txt file is simple but powerful — and easy to mess up. Here are some typical mistakes we’ve seen in audits:
Once you’ve created or updated your robots.txt file, test it using the Google Search Console Robots.txt Tester. This tool shows how Googlebot interprets your rules and flags errors.
Also:
Robots.txt is one of those tools that’s often ignored — until it causes problems. It won’t help your pages rank, but it plays a crucial role in making sure search engines can reach the right content. For businesses focused on performance, visibility and site efficiency, managing crawler access is a smart, low-effort step in any technical SEO strategy. It won’t fix everything, but it keeps your foundation solid — and that’s where long-term growth begins.
When it comes to search engine optimisation, most businesses are focused on ranking pages and getting traffic. But what’s often overlooked is how search engines crawl your site in the first place. That’s where the humble robots.txt file comes in.
It’s a small text file with a big responsibility — directing search engine bots on what to crawl and what to skip. When used properly, it can save crawl budget, tidy up how your site appears in search results, and stop unnecessary load on your server. Used incorrectly, it can quietly block your entire site from search engines.
Let’s break down what this file does, why it matters for SEO, and how to use it wisely.
The robots.txt file sits in the root directory of your website (e.g. https://www.example.com/robots.txt) and acts like a signpost for search engine crawlers. It tells them which areas of your site they shouldn’t crawl.
But here’s the catch: robots.txt only controls crawling, not indexing. That means a page blocked by robots.txt can still appear in search results — without any content — if it’s been linked to from elsewhere.
Also, not all bots respect robots.txt rules. Major ones like Googlebot do, but malicious crawlers usually don’t.
Here are the most common commands used in a robots.txt file:
Important: A Disallow does not stop a page from being indexed. If the page is linked to from elsewhere, it can still appear in search results. To remove it from indexing, you need a noindex meta tag on the page (and that only works if the page is crawled!).
Not every site has a robots.txt, and for small websites, it’s often not critical. But for larger sites — especially in e-commerce, finance, real estate or publishing — it becomes essential for managing crawl behaviour.
Here are four key things to consider:
You can check by visiting yourdomain.com/robots.txt. If it returns a 404, then there is no file. That’s fine by default — search engines will try to crawl everything — but you’re missing the opportunity to control that process.
While User-agent: * is common and applies to all bots, some advanced strategies involve targeting specific ones like Googlebot, Bingbot, or even niche crawlers used by SEO tools.
This can help if you want to treat bots differently — for example, allow Google full access but restrict a third-party crawler.
The format must be precise. A misplaced slash or incorrect directive can cause major SEO issues. For example:
Disallow: / → blocks the entire site.
Disallow: /shop/ → blocks only the /shop/ section.
Allow: /shop/product-page → allows this specific page even if /shop/ is disallowed.
Also check that your robots.txt file returns a status code of 200. A misconfigured server might accidentally return a 403 (forbidden) or 404 (not found), which means search engines will ignore your rules entirely.
Sites with faceted navigation (like /shoes?colour=red&size=10) can end up wasting crawl budget on near-identical pages. You can use robots.txt to block certain patterns or directories — but be cautious. Over-blocking can lead to poor indexing of important content.
Robots.txt is useful in a variety of scenarios. Some of the most common include:
The robots.txt file is simple but powerful — and easy to mess up. Here are some typical mistakes we’ve seen in audits:
Once you’ve created or updated your robots.txt file, test it using the Google Search Console Robots.txt Tester. This tool shows how Googlebot interprets your rules and flags errors.
Also:
Robots.txt is one of those tools that’s often ignored — until it causes problems. It won’t help your pages rank, but it plays a crucial role in making sure search engines can reach the right content. For businesses focused on performance, visibility and site efficiency, managing crawler access is a smart, low-effort step in any technical SEO strategy. It won’t fix everything, but it keeps your foundation solid — and that’s where long-term growth begins.
