What Should You Put in Robots.TXT

You may probably have heard about robots.txt file. And maybe wondering about it. Several questions might be popping up in your head. What is robots.txt? What I should Put in Robots.txt? What is its use? Should I be doing this? Is it necessary? And many more. Answers to all of these questions are probably here.

Robots.TXT file is located in your site’s root. It helps bots like Google to find your great content. And also crawl it. It guides bots What to index and What not to. You can use it to disallow bots to crawl secured, private or any file. Almost all bots read robots.txt before crawling a site.

I have created this post just to answer all of these questions. I’ll probably be helpful. So, STAY TUNED.

What Does Robots.TXT Mean?

When the crawlers known as bots visit your site, first thing they do is visit yoursite.com/robots.txt

Using this file, site owner can tell the crawlers like Googlebot what to index what not to.

You can specify specific files on your site, not to be indexed by bots.

For example, you may want not to index author profile pages using noindex tag.

Here is where robots.txt comes.

You can also tell bots about your sitemap address (which contains a list of articles published on your site). So that, bots can easily find your content.

You can also use it if you don’t want specific file types to be indexed i.e images or anything.

(Photo)

Several questions will be popping up in your head.

Here is a list of questions which you might be asking yourself:

  • Why is Robots.TXT Important?
  • Do You Need A Robots.TXT File?
  • Where Should Robots.TXT Be Located?
  • What Should You Put in Robots.TXT?
  • Robots.TXT Generator
  • Robots.TXT Validator
  • Robots.TXT Examples

I will be answering below all of these questions one by one.

Why is Robots.TXT Important?

Robots.txt file is very much important because all bots first check this file, before crawling your site content and pages.

That’s why you should consider and take some time to put this file in root of your site.

It will help robots understand your site easily and may save your from spam, by preventing indexing of pages like authors.

It will also help robots to easily find your content, so that they can index it.

I don’t think, it can increase your site traffic, by it could be a thing which Google might be checking during AdSense Approval.

(Photo)

So, consider using it.

Do You Need A Robots.TXT File?

If your site is on internet and online (not on some personal server for personal use), you should be having robots.txt file.

As I already mentioned, first thing which bots do is check robots.txt file.

They know from it, What they are to crawl and index and what not to.

It also aids them in finding your content easily on your site (putting sitemap address in it.)

Sitemap contains all the published content & pages URLs of your site.

Where Should Robots.TXT Be Located?

Robots.txt is put in the root of your site.

It should be accessible through yoursite.com/robots.txt

Just make sure you have given read permission of robots.txt file.

If it’s not readable by anyone, it will be useless.

And there will be no use of it.

So make sure to check file read permissions.

What Should You Put in Robots.TXT?

To get an idea you can check our site robots.txt file. Or you can check example section of this article.

Using robots.txt in the wrong way may cause decrease in site traffic.

So, make sure what you are doing.

This is what our site robots.txt file look like:

User-agent: * 
Crawl-Delay: 20
Disallow: /wp-admin/*
Allow: /wp-admin/admin-ajax.php
Noindex: /?*s=
Noindex: */page/*
Sitemap: https://usmanswift.com/sitemap_index.xml
Sitemap: https://usmanswift.com/post-sitemap.xml

Here user-agent: * specifies all bots, not only Google, Bing etc.

Crawl-delay tells about how much time bots should wait to crawl next page.

It helps to decrease server load.

Noindex tag says that bots are not allowed to index the following pages.

You can also include your site sitemap in the robots.txt file as we have and many other people do.

Robots.TXT Generator

You should not be using some kind of tool to generate robots.txt file

People try to save time and do everything by using software and computers.

But it is a sensitive thing.

Suppose if generator puts something which prevents bots to crawl pages, you will lose traffic as I did once on my other site.

So, be patient and give some time manually create your site robots.txt file.

Robots.TXT Validator

You can use Google Webmaster tool to validate your created robots.txt file located at yoursite.com/robots.txt

It will allow you to check that which URLs Google is allowed to access and index.

In this way, it will get easier for you to identify.

If the is a tag containing Noindex, you should test those URLs first in the case if you want Google to index similar pages.

I hope it helped you understanding the use of robots.txt and how can you use it.

If you need any help, let me know in the comments below.

Robots.TXT Examples

Explanation

Conclusion

Robots TXT

Use robots.txt to tell bots what to index and what not to.

Provide sitemap link in the robots.txt

Disallow /wp-admin/ to decrease server load and save bots time to crawl useless pages.

Use Google Webmaster Tool to test final robots.txt created file.

Make sure you can access robots.txt file by visiting yoursite.com/robots.txt

Here is what John Mueller from Google Webmaster team says.

FAQs Summary

Where Should Robots.TXT Be Located?

Robots.TXT file should be in the root of your website i.e https://example.com/robots.txt Make sure you turned on read permission and it is accessible.

What Does Robots.TXT Mean?

Robots.TXT is a file which all crawlers like GoogleBot visit and check before crawling your site You can guide bot what they should and should not index. It also helps Bots to fix content on your site.

Why is Robots.TXT Important?

Robots.TXT is the file which Bots check when they visit any site on the internet. It helps them find content easily. They also come to know that what they are allowed to crawl and index.

Do You Need A Robots.TXT File?

Probably Yes! All websites on the internet should be having robots.txt file. It helps bots to find content easily. They also get what they are allowed to fetch and crawl.

What Should You Put in Robots.TXT?

You should be putting sitemap address in Robots.TXT file. You can also use index and noindex tags to let Bots know what they are allowed to index.

You Might Also Like

LEAVE A REPLY

Please enter your comment!
Please enter your name here