Robots.txt Best Practices: 10 Rules for SEO

Use robots.txt to manage which URLs crawlers can request, not to keep pages out of Google. Google says robots.txt is mainly for avoiding server overload and is not a way to keep a web page out of search; use noindex or a password for that.

Below are 10 rules and the mistakes that do real damage. Check your file, and see which AI bots you block, with the AI Crawler Checker.

What Is Robots.txt?

Robots.txt is a text file websites use to guide search engine crawlers on how to interact with their pages. Stored in the root directory of a website, it contains specific instructions for web crawlers, indicating which areas can be accessed and indexed and which should be restricted.

For instance, a website can block crawlers from accessing admin areas or duplicate pages. While robots.txt directives guide compliant crawlers, they don’t enforce absolute restrictions, as malicious bots may ignore them.

What Is a Robots.txt File Used For?

Control Web Crawler Access

The primary purpose of the robots.txt file is to regulate the behavior of search engine crawlers. By specifying specific rules, website owners can block access to irrelevant or sensitive sections, such as administrative pages, login portals, or development areas.

Prevent Duplicate Content Indexing

Robots.txt can stop crawlers requesting near-duplicate URLs, but it does not consolidate them. Google says not to use robots.txt for canonicalization, because disallowed URLs can still be indexed without their content. Use rel=”canonical” or redirects instead.

Conserve Server Resources

Crawlers consume bandwidth when accessing a site. Google says blocking URLs does not shift crawl budget to other pages unless Google is already hitting your site’s crawl capacity limit.

Protect Sensitive Data

Although robots.txt isn’t a security tool, it cannot keep a URL out of search or protect a file. Use a password or other server-side access control for private files and staging sites.

Facilitate Page Visibility Management

Why Is Robots.txt Important For SEO?

Optimizing Crawl Budget

Google says crawl budget matters mainly for very large or fast-changing sites, roughly 1 million+ pages that change weekly or 10,000+ pages that change daily. Smaller sites rarely need to manage it with robots.txt.

Preventing Indexation of Duplicate Content

Robots.txt is the wrong tool for consolidating duplicates. Use redirects or rel=”canonical”, and keep the duplicates crawlable so Google can see them.

Enhancing Site Security

Robots.txt is not a security measure, and anyone can read it at /robots.txt. Protect sensitive areas with authentication.

Managing Resource-Intensive Content

You can block unimportant image, script or style files, but only if pages loaded without them are not significantly affected. If a resource helps Google understand the page, do not block it.

Guiding Search Engines to Preferred Content

Robots.txt only says what not to crawl. To point crawlers at preferred pages, use internal links and a sitemap.

Preventing the Indexation of Non-Public Pages

Blocking crawling does not prevent indexing. For pages that must stay out of search, use noindex or a password.

Facilitating Smoother Site Migrations and Redesigns

At launch, check the file. A leftover blanket Disallow rule from a staging site blocks crawling of the whole new site.

Aligning with Best Practices for SEO

How Does a Robots.txt File Work?

1. Crawlers Accessing Robots.txt

When a web crawler visits a website, it typically starts by checking the robots.txt file to determine which sections of the site it is allowed to crawl. The file is located at the root of the website’s domain (e.g., https://www.example.com/robots.txt). If the robots.txt file is absent, crawlers assume that all parts of the site are open for crawling.

1. Syntax and Directives
   The robots.txt file uses specific directives to communicate with crawlers:
   • User-agent: Specifies the targeted web crawler. An asterisk (*) denotes all crawlers.
   • Disallow: Indicates the URLs or directories that should not be crawled.
   • Allow: Permits access to specific subdirectories or pages, even within a disallowed directory.
   • Crawl-delay: Sets a delay between requests for some crawlers. Google does not support it.
   • Sitemap: Provides the location of the website’s sitemap to assist crawlers in finding all accessible pages.

For example, to prevent all crawlers from accessing the /private directory while allowing them to crawl everything else, the robots.txt file would include:

3. Crawlers Interpreting Robots.txt

Compliant web crawlers read the robots.txt file and adhere to its directives, avoiding disallowed areas. However, it’s important to note that the robots.txt file operates on a voluntary compliance basis. While reputable search engines like Google and Bing follow these instructions, malicious bots may ignore them, underscoring that robots.txt is not a security measure but a guideline.

10 Robots txt Best Practices

1. Create a Robots.txt File in the Root Directory

This text file must be placed in the root directory of your website (e.g., https://example.com/robots.txt) to be accessible to web crawlers. You can create it using any text editor like Notepad or a code editor, saving it as “robots.txt” with a lowercase “r” to ensure compatibility.

The file consists of simple directives like User-agent to specify which crawler the rule applies to and Disallow to block specific pages or directories. For example:

This setup tells all crawlers to avoid the /private/ folder.

2. Use the Correct Syntax for User-Agent and Disallow Rules

The file uses two main elements: User-agent and Disallow.

• User-agent: Specifies which bot the rule applies to (e.g., Googlebot, Bingbot). Use an asterisk (*) to target all crawlers.
• Disallow: Lists the directories or pages you want to block from being crawled.

Example:

This tells all crawlers to avoid the /admin/ folder. Use Allow to permit crawling within a disallowed section. Google ignores Crawl-delay.

Check the file and the bots: the AI Crawler Checker reads your live robots.txt and shows which search and AI crawlers are allowed or blocked.

3. Test the File for Errors Using Google Search Console

Google Search Console includes a robots.txt report for this purpose.

To check your file, open Search Console > Settings > robots.txt report. It shows which robots.txt files Google found, when they were last crawled, and any warnings or errors. Developers can test rules locally with Google’s open-source robots.txt library.

4. Ensure the File Is Easily Accessible at /robots.txt

The standard location for this file is directly at https://example.com/robots.txt. If the file isn’t in this exact location, most crawlers won’t find it, and your directives won’t be applied.

Keep in mind that the robots.txt file is case-sensitive. Using incorrect capitalization, such as Robots.TXT or placing it in a subdirectory, will render the file ineffective.

It’s also important to regularly verify the file’s accessibility by visiting its URL in a browser or using testing tools like Google Search Console.

Confirm important pages are indexed: paste a key URL into the Google Index Checker to see whether Google has it.

5. Avoid Blocking Essential Content Accidentally

Accidentally blocking important content in your robots.txt file can harm your site’s SEO and visibility in search engine results. To prevent this, carefully review the pages and directories listed under Disallow. Ensure that critical pages such as your homepage, product pages, or key blog posts are not restricted.

For example, avoid overly broad directives like:

This blocks the entire site from being crawled. Instead, use precise rules to target specific sections you want to exclude, such as:

Test your robots.txt file regularly using tools like Google Search Console to identify misconfigurations.

6. Use Specific Directives for Multimedia Resources

Managing how search engines interact with multimedia resources, such as images, videos, or PDFs, is crucial for optimizing crawl efficiency. Use robots.txt to block these resources if they are unnecessary for search visibility or consume excessive crawl budget.

For example, to prevent crawlers from indexing an images folder:

Blocking multimedia can reduce server load and prevent irrelevant content from appearing in search results.

7. Limit Crawl Budget Wastage by Targeting Irrelevant Pages

Crawl budget is the number of pages search engines crawl on your site within a given timeframe. To optimize this resource, use robots.txt to block access to irrelevant or low-value pages that don’t contribute to your SEO goals.

Examples of such pages include:

• Admin panels (/admin/)
• Login pages (/login/)
• Search results pages (/search/)

For instance:

Avoid wasting crawl budget on unnecessary pages by identifying irrelevant sections and updating your robots.txt file regularly.

8. Use Meta Tags for Page-Level Blocking When Possible

While robots.txt is effective for managing crawler access, it’s not always the best option for blocking specific pages. For page-level exclusions, using a noindex meta tag is often more precise and reliable. This ensures the page remains accessible to users while signaling search engines not to index it.

For example, to block indexing of a specific page:

This approach is beneficial when you want to allow crawlers to follow links on a page but exclude the page itself from search results. Unlike robots.txt, meta tags work directly at the page level, providing more granular control. Do not block a page in robots.txt if you need Google to see its noindex tag; Google warns that combining crawling and indexing rules can make them cancel each other out.

9. Keep the Robots.txt File Updated with Site Changes

Regularly updating your robots.txt file ensures it remains aligned with your website’s structure and SEO strategy. When you add new pages, sections, or directories, revisit the file to check if any adjustments are needed. Similarly, if outdated pages or areas have been removed, clean up their references in robots.txt to avoid confusion.

For instance, if you add a /blog/ section, ensure that it’s not unintentionally blocked:

Outdated or inaccurate directives can lead to search engines missing important content or wasting crawl budget. Schedule periodic reviews of your robots.txt file, particularly during major site updates or migrations, to maintain a seamless and effective crawling experience for search engines.

10. Avoid Revealing Sensitive Paths Unnecessarily

While robots.txt can block crawlers from accessing sensitive directories, it’s important not to disclose sensitive or confidential paths unnecessarily. Since robots.txt is publicly accessible at /robots.txt, anyone can view the file, including malicious bots or users.

For example, avoid explicitly listing paths like:

Instead, secure sensitive areas through server-side methods such as password protection or IP whitelisting. Use robots.txt to block non-sensitive but unnecessary content, like internal search results or duplicate pages.

How To Create A Robot.txt File?

Understanding the Basics

The robots.txt file resides in your website’s root directory (e.g., www.example.com/robots.txt). It uses a straightforward syntax to communicate with web crawlers:

• User-agent: Specifies the crawler to which the rule applies.
• Disallow: Denies access to specified directories or pages.
• Allow: Grants access to specific directories or pages (used primarily to override a disallow rule).

Step-by-Step Guide to Creating a robots.txt File

1. Identify the User-Agent
   Determine which crawlers you want to address. For instance, Google’s crawler is Googlebot, while Bing’s is Bingbot. To apply rules universally, use the asterisk (*) wildcard:

2. Specify Directories or Pages to Disallow

Decide which parts of your site you want to prevent crawlers from accessing. For example, to block all crawlers from accessing your site’s admin area and a specific private page:

3. Allow Specific Access

If you have a directory generally disallowed but want to permit access to a particular file within it, you can specify:

4. Include Your Sitemap

Facilitate efficient crawling by specifying the location of your sitemap:

Practical Example

Imagine you run an online store with the following structure:

• /admin/
• /user-data/
• /products/
• /products/sale/
• /checkout/

You want to:

• Prevent crawlers from accessing administrative and user data directories.
• Allow crawlers to index all products, including those on sale.
• Block the checkout process pages.

Your robots.txt file would look like:

5. Testing Your robots.txt File

Before deploying, it’s crucial to test your robots.txt file to ensure it behaves as intended. Search Console’s robots.txt report shows which robots.txt files Google found, when they were last crawled, and any warnings or errors.

How to Find a Robots.txt File?

Finding a website’s robots.txt file is straightforward, as it is always located in a specific location for search engines to easily access. Here are several ways to locate a robots.txt file:

1. Direct URL Access

The robots.txt file is always located in the root directory of a website. To find it, simply append /robots.txt to the website’s domain name. For example:

Enter this URL in your browser’s address bar, and the robots.txt file will display if it exists. If the file is missing, you’ll receive a “404 Not Found” error.

2. Using Browser Developer Tools

Modern browsers have developer tools that allow you to check for the robots.txt file. Here’s how:

• Open the website in a browser.
• Right-click anywhere on the page and select Inspect.
• Look under the Network tab while refreshing the page. Search for the robots.txt request to confirm if the file is present and accessible.

3. Using Online Tools

Several online tools can help locate and analyze a website’s robots.txt file:

• Google Search Console: If you have access to the site’s account, open Settings and the robots.txt report.
• Robots.txt Testers: Tools like Ahrefs, SEMrush, or Screaming Frog can quickly identify the file and test its configuration.

4. Command-Line Methods

For developers, command-line tools like curl or wget can fetch the robots.txt file:

This method downloads the file to your local system for review.

5. Search Engine Archives

If a website’s robots.txt file has been indexed previously, you may find a cached version on Google. Use search queries like:

Run a wider check: the Website SEO Audit flags crawl and indexing problems around robots.txt, sitemaps and canonicals.

Common Robots.txt Mistakes to Avoid

Here are key errors to avoid:

1. Incorrect Placement of the Robots.txt File

The robots.txt file must reside in your website’s root directory (e.g., https://www.example.com/robots.txt). Placing it elsewhere prevents crawlers from locating it, rendering your directives ineffective.

2. Misuse of Wildcards

While wildcards like * (representing any sequence of characters) and $ (indicating the end of a URL) can simplify directives, improper use may unintentionally block essential content. For instance, Disallow: /*.php$ blocks all .php pages, which might include critical pages. Use wildcards cautiously and test directives to ensure they perform as intended.

3. Using Deprecated ‘Noindex’ Directive

As of September 2019, Google no longer supports the noindex directive in robots.txt. Relying on this method will not prevent pages from appearing in search results. Instead, implement the noindex meta tag within the HTML of the specific pages you wish to exclude from indexing.

4. Blocking Essential Resources

Disallowing access to directories containing CSS, JavaScript, or image files can hinder search engines from rendering your site correctly, leading to misinterpretation of content and potential ranking issues. Ensure that necessary resources are accessible to crawlers to maintain proper site functionality.

5. Omitting Sitemap URL

Including the location of your XML sitemap in the robots.txt file assists crawlers in discovering and indexing your site’s pages more efficiently. Add a line such as Sitemap: https://www.example.com/sitemap.xml to guide search engines.

6. Allowing Access to Development or Staging Sites

Permitting crawlers to index development or staging environments can lead to duplicate content issues and unintended exposure of unfinished content. Password-protect them; robots.txt alone does not keep a URL out of search.

7. Using Absolute URLs Instead of Relative Paths

In robots.txt, directives should use relative paths rather than absolute URLs. For example, use Disallow: /private/ instead of Disallow: https://www.example.com/private/. This ensures directives are correctly interpreted across different protocols and subdomains.

8. Utilizing Deprecated or Unsupported Elements

9. Blocking Canonicalized URLs

If URLs specified in canonical tags are blocked in robots.txt, search engines cannot access them to see the canonicalization, leading to potential duplicate content issues. Ensure that canonical URLs are accessible to crawlers.

10. Ignoring Case Sensitivity

The robots.txt file is case-sensitive. Directives like Disallow: /Private/ and Disallow: /private/ are interpreted differently. Consistently use the correct case to ensure directives apply as intended.

Final Discussion

Keep the file short, test it after every change, and use noindex or a password for anything that must stay out of search.

FAQs

What is a robots.txt file used for?

A robots.txt file tells crawlers which URLs they may request on your site. Google says it is used mainly to avoid overloading your server and to skip unimportant or similar pages. It is not a way to hide pages from search results.

Does robots.txt stop a page appearing in Google?

No. If other pages link to a blocked URL, Google can still index the URL without visiting it, showing a result with no description. To keep a page out of search, use a noindex tag or password protection, and leave the page crawlable so Google can see the tag.

Does Google support crawl-delay in robots.txt?

No. Google’s robots.txt documentation lists only user-agent, allow, disallow and sitemap as supported fields and says others, such as crawl-delay, are not supported. Some other crawlers may honour crawl-delay, but you cannot use it to slow Googlebot.

How do I check my robots.txt file?

Open yourdomain.com/robots.txt in a browser on the exact host, protocol and port you want to check. In Search Console, the robots.txt report shows which files Google found, when they were last crawled and any warnings or errors. Developers can test rules locally with Google’s open-source parser.