|
|
|
Bring Your Web Site to Life With PHP
PHP: Hypertext Preprocessor, better known as PHP, is a highly popular, server-side scripting language that can be embedded directly into HTML coding. PHP can do anything that CGI can do, such as process form data and auto generate dynamic content,...
Dead domains - how to give them a decent burial
If you've registered a domain name, built a web site and have had visitors but no sales, you might feel like abandoning the site and the domain. Find out why there is value in your domain names and what to do about it.
How many...
Finding the Right Web Hosting Plan
There are many things to consider when choosing a web hosting
plan. The five most important are listed below.
1. Disk Space/Storage
2. Data Transfer
3. Cost
4. Programming Languages
5. Support
1....
How Search Engines Work
Before anyone can start optimizing a web site, you must understand how search engines work.
Search engine optimization is the hardest thing to do for a webmaster because there are so many rules to it and you have to stay up to date with all...
Why Autoresponders Are an Essential Tool for Every Internet Business
I admit it -- when I'm on the Internet, I want everything instantly. And I know I'm not alone; in our high-speed world we expect to get what we want with the click-of-a- mouse. To help fulfill this desire for everything "now, now, now!", many...
|
|
| |
|
|
|
|
|
|
The proper way to use the robots.txt file
When optimizing your web site most webmasters don?t consider using the robots.txt file. This is a very important file for your site. It let the spiders and crawlers know what they can and can not index. This is helpful in keeping them out of folders that you do not want index like the admin or stats folder or content that they can not index.
Here is a list of variables that you can include in a robots.txt file and there meaning:
1)User-agent: In this field you can specify a specific robot to describe access policy for or a ?*? for all robots more explained in example.
2)Disallow: In the field you specify the files and folders not to include in the crawl.
3)# the number sign represents comments
Here are some examples
of a robots.txt file for redball.com
User-agent: *
Disallow:
The above would let all spiders index all content.
Here another example
User-agent: *
Disallow: /cgi-bin/
The above would block all spiders from indexing the cgi-bin directory.
User-agent: googlebot
Disallow:
User-agent: *
Disallow: /admin.php
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /stats/
In the above example googlebot can index everything while all other spiders can not index admin.php, cgi-bin, admin, and stats directory. Notice that you can block single files like admin.php.
About the Author
Jimmy Whisenhunt is the owner of VIP Enterprises
|
|
|
|
|
|
