The robot text file is used to disallow specific or all search engine spider’s access to folders or pages that you don't want indexed.

Why would you want to do this?

You may have created a personnel page for company employees that you don't want listed. Some webmasters use it to exclude their guest book pages so to avoid people spamming. There are many different reasons to use the robots text file.

How do I use it?

You need to upload it to the root of your web site or it will not work - if you don't have access to the root then you will need to use a Meta tag to disallow access. You need to include both the user agent and a file or folder to disallow.
What does it look like?
It's really nothing more than a "Notepad" type .txt file named "robots.txt"
The basic syntax is
User-agent: spiders name here
Disallow:/ filename here
If you use
User-agent: *
The * acts as a wildcard and disallows all spiders. You may want to use this to stop search engines listing unfinished pages.
To disallow an entire directory use
Disallow:/mydirectory/
To disallow an individual file use
Disallow:/file.htm
You have to use a separate line for each disallow. You cannot you for example use Disallow:/file1.htm,file2.html You should use
Use-agent/*
Disallow:/file1.htm
Disallow:/file2.htm

For a list of spider names visit

http://www.robotstxt.org/wc/active/html/
Make sure you use the right syntax if you don't it will not work. You can check you syntax here http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

For help on creating robot text files there is a program call robogen.
There is a free version and an advanced version, which costs $12.99 http://www.rietta.com/robogen/ Alan Murray is a Certified Internet Webmaster Professional and Provides SEO Services and Website design.

http://www.designprofessional.co.uk/SEO-Services.htm

Search Engine Optimization Articles

• SEO Case Study: Attorneys Chasing Vioxx Organic Search Ranking posted Nov. 14, 2004
• How A Change in Perspective Can Help You Make More Holiday Season Sales: Pt. One posted Nov. 29, 2004
• Do Not Drop Your Web Site Off the Search Engine Cliff posted Feb. 15, 2005
• Creating Robots.txt File and its Importance posted Feb. 18, 2005
• Search through Google and see for yourself! posted June. 8, 2005
• Choosing a web hosting plan posted Jan 16, 2006
• Why Good Looks Will Always Give You More Traffic- posted Jan 16, 2006
• How To Use The Net To Climb Much Faster Up The Corporate Ladder- posted Jan 16, 2006
• How To Create A Successful Forum- posted Jan 16, 2006
• SEO- 10 Costly Search Engine Mistakes to Avoid- posted Jan 17, 2006
• Search Engine Optimization Articles, Googlebot Won't Go Home. posted Nov. 6, 2004
• Increase Your Page Rank Through SEO posted Nov. 6, 2004
• You Attract Who You Are On The Internet! posted Nov. 6, 2004
• SOME COPY TIPS FROM AN OLD HAND posted Nov. 9, 2004
• Why is search engine Optimization expensive? posted Nov. 9, 2004
• The Reality of Search Engine Submissions posted Nov. 9, 2004
• How to get top position rankings in the search engines every time within 28 days! posted Nov. 14, 2004
• Get listed at DMOZ posted Nov. 29, 2004
• Dear Bill Gates, You Clever Fox posted Nov. 29, 2004
• Optimize your Search Engine Placement Five Easy Ways posted Nov. 29, 2004
• There Should Be More to your SEO Consultant Than Rankings posted Feb. 15, 2005
• Optimizing Your Website's Design For Search Engines posted Feb. 18, 2005
• Time to say goodbye to submission software? - posted Jan 16, 2006
• Extra! Extra! Knowledge Is The Ultimate Power Online...- posted Jan 16, 2006
• Jagger Algo Update - Part 1- posted Jan 16, 2006
• Jagger Algo Update - Part 2- posted Jan 16, 2006
• Pay Per Click Vs Natural Listings- posted Jan 20, 2006
• All This Talk about Link Building- posted Jan 20, 2006