Robots.txt
Robots.txt play an vital role when it comes to on Page SEO, It passes an instruction to search engines about which page should landing page should crawl, index and which page shouldn’t.
So today let’s try to understand every minute detail about Robots.txt optimization in SEO and how it can help to boost our on page game in 2024.
- What is Robots.txt?
- Benefits of Robots.xtx
- Why is it important for SEO?
- How to create Robots.txt?
- How to apply robots.txt to websites?
- Most common mistakes to avoid in robots.txt
What is Robots.txt?
Robots.txt is a text file that is available in the webserver, passes instructions to the search engines about what landing pages should be crawled, indexed and shouldn’t.
Syntax:
Formula:
- User-agent: *
- Allow: /
- Disallow: /
Benefits of Robots.txt File for SEO
- Block & Unblock: Robots.txt helps to block and unblock unnecessary pages from search index, so that our crawl budget can be saved.
- Improved Crawling Efficiency: It actually helps to balance the crawl budget, and efficient usage. So that our crawl budget will not get wasted unnecessarily.
- Protection of Sensitive Info: we can protect our sensitive information such as private content which we don’t want to be visible in the Search engine Results page.
- Prevention of Duplicate Content Issues: It helps prevent search engines from indexing multiple versions of the same content from SERP.
How to create Robots.txt?
There are different ways of robots.txt creations, some of the most commonly used practices are listed below.
Syntax of Robots.txt
- User-agent: *
- Allow: /
- Disallow: /
Let’s try to understand the basic syntax of robots.txt file
- User-Agent: to which web crawler you would like to communicate
- Allow: what pages of your site is allowed for indexing and crawling
- Disallow: what pages of your site are not allowed for indexing and crawling
Example: we wanted to specific pages of a website and those pages are listed below
- blogs.janardhan.digital/old-seo-on-page
- blogs.janardhan.digital/old-google-ads
Syntax of Robots.txt for the above example
- User-agent: *
- Allow: /
- Disallow: /old-seo-on-page
- Disallow: /old-google-ads
We need to pass the path of the page instead of the whole site, so that it can block specific pages
Disallowing Specific Directories:
Syntax
- User-agent: *
- Disallow: /restricted-directory/
Allowing Crawling of Specific Directories:
- User-agent: *
- Disallow: /private/
- Allow: /private/public/
Blocking Specific Web Crawlers:
Syntax
- User-agent: Googlebot
- Disallow: /no-googlebot/
Crawl Delay: it can slow down the crawl rate for specific user agents. This syntax can be used to reduce server load on our website.
Syntax
- User-agent: *
- Crawl-delay: 5
Sitemap Declaration:
It is very common practice to include a sitemap with the Robots.txt file to inform the number of pages available in the whole website. Web crawler will take all of these pages into consideration and block the pages mentioned with disallowed commands.
- User-agent: *
- Disallow: /
- Allow: /
Sitemap: blogs.janardhan.digitl/sitemap.xml
How to apply robots.txt to websites?
Robots.txt file can be applied to website in multiple ways
- Through file manager on hosting manager
- Through Yoast SEO plugin on wordpress
Most common mistakes to avoid in robots.txt
There are some of the most common mistakes to avoid in the robots.txt file, a couple of them are mentioned below.
- Blocking Important Pages: we should be very careful when it comes to passing disallow commands in the robots.txt file.
- Syntax Errors: we should not misspell or avoid the syntax characters as for the standard format.
- Disallowing the Entire Website: this is one of the rare case mistakes that can happen when we implement. Ideally We should not block the entire site from indexing.
- Blocking Essential Crawlers: we should be very about what crawler is important for our business model, what is not and block accordingly.
- Ignoring Case Sensitivity: The robots.txt file exclusion protocol is case-sensitive.
- Using Disallow with No User-Agent: we should not keep it empty
- Sitemap file: we should not update the robots.txt file without sitemap file
How to Check status of Robots.txt Updation
- domainname.com/robots.txt
Example:
- Blogs.janardhan.digital/robots.txt
