What Should You Put in Robots.TXT

Robots.TXT file is located in your site's root. It helps bots like Google to find your great content. And also crawl it. It guides bots What to index and What not to. You can use it to disallow bots to crawl secured, private or any file. Almost all bots read robots.txt before crawling a site.

Recent posts


Popular categories