Untitled Document

Default Robots.txt file setting;

Default - All Robots are:

Crawl-Delay:

Sitemap: (leave blank for none)
example: http://www.yoursite.com/sitemap.xml

Specific Search Robots;

Search Engine	Choose Setting	Bot-Name
Google		GoogleBot
Bing		BingBot
Yahoo		Yahoo-slurp
Alexa/Wayback		ia_archiver
Ask/Teoma		Teoma *deprecated
Yandex		Yandex
GigaBlast		Gigabot
Scrub The Web		Scrubby
Baidu		Baiduspider
DuckDuckGo		DuckDuckBot
Blekko		ScoutJet

Specific Special Bots;

Search Engine	Choose Setting	Bot-Name
Google Image		googlebot-image
Yahoo MM		yahoo-mmcrawler *deprecated
MSN Pic-Search		psbot
Yahoo Blogs		yahoo-blogs/v3.9 *deprecated

Extra Exclusion features;

Search Engine	Choose Setting	Bot-Name
Google		GoogleBot
Bing		BingBot
Yahoo		Yahoo-Slurp
Baidu		Baiduspider
Yandex		Yandex
DuckDuckGo		DuckDuckBot
Blekko		ScoutJet

Disallow Bad Bots;
Select this option to block the most common "Bad-Bots," also known for spam and email harvesting from accessing your domain.
Disallow Bad-Bots:	User-agent: BotALot User-agent: BuiltBotTough User-agent: BunnySlippers User-agent: Cegbfeieh User-agent: CheeseBot User-agent: CherryPicker User-agent: CopyRightCheck User-agent: cosmos User-agent: Crescent User-agent: DittoSpyder User-agent: EmailCollector User-agent: EmailSiphon User-agent: EmailWolf User-agent: EroCrawler User-agent: ExtractorPro User-agent: Foobot User-agent: hloader User-agent: httplib User-agent: humanlinks User-agent: InfoNaviRobot User-agent: JennyBot User-agent: LexiBot User-agent: LinkextractorPro User-agent: LinkWalker User-agent: LNSpiderguy User-agent: lwp-trivial User-agent: MIIxpc User-agent: moget User-agent: NetAnts User-agent: NICErsPRO User-agent: Openfind User-agent: ProWebWalker User-agent: RepoMonkey User-agent: RMA User-agent: SiteSnagger User-agent: SpankBot User-agent: spanner User-agent: suzuran User-agent: Teleport User-agent: TeleportPro User-agent: Telesoft User-agent: TheNomad User-agent: TightTwatBot User-agent: Titan User-agent: True_Robot User-agent: turingos User-agent: VCI User-agent: WebAuto User-agent: WebBandit User-agent: WebCopier User-agent: WebEnhancer User-agent: WebmasterWorldForumBot User-agent: WebSauger User-agent: WebStripper User-agent: WebZip User-agent: Wget User-agent: WWW-Collector-E User-agent: Xenu's User-agent: Zeus

Restricted Directories;

The path is relative to root and must contain a trailing "/"

Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"
Exclude Directory: "/"

Create The Robots.txt;

Copy and paste the text above into notepad and save it as robots.txt.
Upload your robots.txt file into your root directory.

Example: http://www.yoursite.com/robots.txt