Robots.TXT file is located in your site's root. It helps bots like Google to find your great content. And also crawl it. It guides bots What to index and What not to. You can use it to disallow bots to crawl secured, private or any file. Almost all bots read robots.txt before crawling a site.