How To Use The Robots.txt File To Increase Your Web Ranking




Typically we tend to rank well on one engine for a particular keyphrase and assume that each one search engines will like our pages, and hence we tend to will rank well for that keyphrase on a range of engines. Sadly this can be rarely the case. All the majorsearch engines differ somewhat, thus what’s get you ranked high on one engine might truly facilitate to lower your ranking on another engine.
It’s because of this that some folks like to optimize pages for each explicit search engine. Sometimes these pages would solely be slightly completely different but this slight difference might build all the difference when it involves ranking high.
However because search engine spiders crawl through sites indexing each page it will notice, it might return across your search engine specific optimizes pages and as a result of they’re very similar, the spider may suppose you’re spamming it and can do one among two things, ban your website altogether or severely punish you in the shape of lower rankings.
The solution is this case is to stop specific Search Engine spiders from indexing some of your internet pages. This is done employing a robots.txt file which resides on your webspace.
A Robots.txt file is a very important half of any webmasters battle against getting banned or punished by the search engines if she or he styles completely different pages for various search engine’s.
The robots.txt file is just a easy text file as the file extension suggests. It’s created using a straightforward text editor like notepad or WordPad, difficult word processors like Microsoft Word will only corrupt the file.
You can insert certain code during this text file to make it work. This is how it will be done.
User-Agent: (Spider Name)
Disallow: (File Name)
The User-Agent is the name of the search engines spider and Disallow is that the name of the file that you do not wish that spider to index.
You have got to start out a brand new batch of code for every engine, but if you wish to list multiply disallow files you’ll one below another. For instance
User-Agent: Slurp (Inktomi’s spider)
Disallow: xyz-gg.html
Disallow: xyz-al.html
Disallow: xxyyzz-gg.html
Disallow: xxyyzz-al.html
The on top of code disallows Inktomi to spider 2 pages optimized for Google (gg) and two pages optimized for AltaVista (al). If Inktomi were allowed to spider these pages likewise as the pages specifically made for Inktomi, you may run the chance of being banned or penalized. Hence, it is often a smart plan to use arobots.txt file.
The robots.txt file resides on your webspace, but where on your webspace? The foundation directory! If you upload your file to sub-directories it will not work. If you needed to disallow all engines from indexing a file, you just use the “*” character where the engines name would sometimes be. However beware that the “*” character will not work on the Disallow line.
Here are the names of a few of the large engines:
Excite – ArchitextSpider
AltaVista – Scooter
Lycos – Lycos_Spider_(T-Rex)
Google – Googlebot
Alltheweb – FAST-WebCrawler
Be certain to check over the file before uploading it, as you’ll have made a simple mistake, that might mean your pages are indexed by engines you don’t need to index them, or maybe worse none of your pages may be indexed.
Are you looking for more information on website seo. Or about website seo. Get pro advice on website seo.
How To Use The Robots.txt File To Increase Your Web Ranking
Mail this post
