Home
About
Apple
Career
Experiences
General
Graphics
Hardware
History
Humor
Interface
Networking
OS
Opinion
Politics
Programming
Quotes
Reviews
Security
Software
Sound
Thought
Web
Cheap International
Airfare Online
Wachovia online banking
Get Free Coupons Online
Finding the perfect
discount hot tub
Payday Loans
Stock Trading Online
Stuffed Animals
Smart Investing Online
|
|
Bots, Crawlers and Spiders, Oh my!
How search engines work...
By: David K. Every
|
Kind:
Created:
Size: |
Article
2002-03-11 08:34:06
4 KB |
|
|
 |
 ost people creating their own websites want their site to be at the top of the search engines response list when someone searches for a particular topic. While that isn't likely, there are things you can do to move up the priority list, and it is good to understand how these things work.
There is something referred to as a bot (robot), crawler or spider. Their purpose is to automatically surf the web. They go to every site, and try to follow every link they can find; then they crawl all over anything they can find there as well; hence the bug references. When you register your site with a search engine, that is all you are doing - telling their spider to start crawling your site; they will find you eventually, whether you register or not.
Search engines employ these (none-too-smart) automatons to look for anything new, and to create an "index" of what they find on each site. They keep a list of topics and key words, and they count them up. If each page on your site has the word "computer" in it, then they can "guesstimate" that your site has something to do with computers. Then when someone searches for the word "computer", they know that your site has something to do with that topic, and you should be in the list of 15,000+ sites that also refer to computers.
The problem is that there are so many sites and pages that have to do with every topic, that they also need to figure out who should be first on the list. They need to figure out popularity and give each site (or page) some relative weight, with the most massive sites showing up higher on the list.
If you want a lot of weight quickly, the search engines will let you rent it (a form of advertising); but most of us don't have the budgets to create that artificial weight, and must do it other ways.
Search engines can do little to figure out true "popularity" and how often people visit; they can't really snoop other people's sites and see who is visiting or how often, so they resort to less direct methods.
On the extreme high end, there is some ability to poll users and figure out where they are going; but you're not likely to show up in those, and we're not talking about creating a site for a company that can measurably effect the nations GDP; just a normal site.
One of the ways that search engines can guess at popularity is to just count links; they look at how many other sites are pointing to a page on your site, then they can rate how "popular" your site is. The more people that point to you, and the bigger they are, then the more valuable your information must be; and the more weight you get. So if you want to show up better on search engines, then you need to make web-friends and link to each other. Advertising banners on other sites (that have weight), don't hurt, since that adds some weight (links and readers); but the banner links stop when you stop advertising.
Another way is just based on how big your site is; if there are many articles on the site (a lot of "content"), that all have articles about the same subject, then you're probably getting visitors on that topic, and others would probably have interest as well.
In fact, content is often the secret to being a successful site. If you create or put up good information (content) on subjects that people have interest in, then you'll rank higher. You can go to other sites that have interest in that topic, and ask them to link to your articles. Others will link to them on their own. Either way, you'll be gaining readers, and "weight" as far as the search engines are concerned; which means others will find your site easier, and they'll be more likely to link to it; which adds even more to your weight. Like a snowball rolling down a hill, the more content you have, the more momentum you pick up, and then the more people notice you. Content and a little self-promotion go a long way in the popularity of a web site.
 Format for Printing  Mail
|
|
|
|
