Robots Text Generator – ReddWebDev

The robots.txt is a very simple text file that is placed on your root directory. An example would be www.yourdomain.com/robots.txt. This file tells search engine and other robots which areas of your site they are allowed to visit and index. Every website on the internet should have a robots.txt file located in the root directory.

You may use the form below to create your own fairly extensive robots.txt file automatically. Simply read and select which bots or options you would want your robots.txt file to contain. When you are finished, simply copy and paste into notepad, name it robots.txt and upload it into the root directory of your domain and you’ll be good to go.

A robots.txt file helps various robots (internet parsing utilities) to know which part of your domain to parse and which parts not to parse. There are hundreds of internet parsing agents, or robots on the world wide web, so it’s important to give direction to those robots that actually do read the robots.txt file.