Robots.txt is an instruction file to specifically give search engine robots the rules that they must follow. Robots instructions can also be written as meta tags into individual pages. A common misconception is that not having a robots.txt will prevent your website from being crawled or discovered by search engine crawlers. This is not true. A crawler such as GoogleBot, will still crawl your site and spider all of your content.
What can get tricky is if you accidentally provide incorrect instructions in your Robots.txt – this can cause the crawler to miss entire areas of your website. Having a correct Robots.txt file available will help limit the off-limits areas of your site for search crawlers.
Some webmasters also make extensive use of the Robots.txt to restrict spam crawlers from accessing the website.
Related posts:
