Straightforward Explanation of Search Engine Data Bases

One of our most frequently asked questions is “why do I have to pay for search engine submissions?” Though the main premise of that question is false, it prompted us to break the search engine topic out of our “traffic” topic to discuss it in more detail.

It is important to understand that in most cases search engines are businesses with employees, payroll, and other financial obligations. We say “most” because there are one or two exceptions among major search engine database providers (specifically But all of these search engines rely on the integrity of their data and whether or not the internet consumer will turn to them when they need information. Ensuring this integrity of data is paramount in getting a grasp on how search engines work.

If you are looking for web page optimization guidelines click here.

There are four major techniques used by search engines to collect and organize web site information for their search programs. Search engines may use either or a combination of some or all data collection methods.

Manual Web Site Review:

One solid philosophy behind compiling a useful database is to have real people review each submission to be sure it is appropriate and useful to the general public. Manual site review also assures that the submitted sites are categorized and indexed properly so that people will be able to find the information they are looking for.

As you can imagine, with thousands of submissions coming in daily, manual review can be a time consuming project requiring knowledgeable staff who understand the internet and how the public will use the data. Unless this process is done by volunteers (as it is at DMOZ), it can be a costly venture but in the end manual review assures that you have highly reliable search data that weeds out sites that are trying to appear to be something that they are not. The main downfall of this method is that a search may not retrieve all of the sites on the internet that pertain to your search criteria since the database only includes sites that have actually been reviewed.

One of the most important free submissions you can make is to (the open directory project). DMOZ is a free service but their database is used as one of the primary sources of web sites by many of the most popular search engines. Before submitting to DMOZ be sure to read their instructions so that you are following their guidelines. Be prepared to wait a few months for your submission to be included because DMOZ is reviewed by real people and there are a great number of sites submitted each day. The timing depends primarilly on whether or not your category has a designated editor and how many sites are being submitted to it.

Robot Site Collection:

On the other end of the spectrum are databases that are entirely compiled and categorized by automated programs. These programs go out from site to site and read web pages, gathering information contained in specific HTML tags and text on the site to determine how to add it to its database. These “robots”, or “spiders” and “crawlers” as they are often called, then follow links on the pages to move on to the next web page or website to be considered.

Though this method is not labor intensive, it can create huge databases that require a good deal of equipment and internet resources. These robot generated databases tend to include more sites than those reviewed manually but relevant sites can often get lost in the number of items returned by a search. Though search engines are constantly upgrading and improving their search programs to minimize returning non-relevant sites to people using their services the automation involved makes this a never ending task. One of the best online resources for keeping up with search engine information is We highly recommend bookmarking this site and becoming familiar with what is happening in the industry.

Automated Categorization:

Another method for compiling a search engine database is the automated categorization of sites that are submitted manually. This method takes most of the manual aspects of site review and hands them over to sophisticated programs designed to combine data collected from the submission entry and data on the submitted web site in an attempt to provide a solid database for the information consumer. Many of these search engines try to minimize the spoofing of their programs with manual random checks on submitted entries and reviews prompted by public responses.

Paid Placement:

There are a number of different variations of paid placement databases and many search engines that use one of the other methods as their primary database collection method also use some form of paid placement. Though some search engines such as Yahoo and Alta Vista display a few paid entries on each results page, they make it clear which are “sponsored” sites and which are generated from relevant general database searches.

On the other hand, other search engines, such as Infoseek and MSN (at the time of this writing) exclusively use paid results (MSN will show unpaid results if no items are found in the paid database). These search engines try to portray their results as “highly relevant” because advertisers bid on keywords. We’ve sent requests for the logic behind that statement but for now it still eludes us.


Express Setup:

A few sites that use the manual review method offer a service that moves your submission to the stop of the review stack. This means that your site is reviewed quicker and usually has no effect on actual placement. We mention this because it can appear that a site is a paid placement site when, as long as they also allow free submissions, it really isn’t.

What does this information do for you?
Search Engine Submission

What to do with this information requires that a merchants decide whether or not they are equipped to locate and submit to search engines or if they should hire a professional to help. We’ve tried to emphasize the importance of staying current on search engine information and because of the difficulty in doing this while maintaining other aspects of your business we highly recommend the hiring of a professional that specializes in Search Engine Optimization (SEO).

It is, however possible to handle this yourself. We’ve reviewed some of the best resources that will give you the information you need. With the right knowledge and a good place to start… the journey can be very rewarding.

Web based resources:

  • Webmaster World discussion boards
    – Anwers to practically any question you might have.
    – Excellent site with step by step answers for doing things right.
    – This site is hard to reach but keep trying because the author does an excellent job.


  • Search Engine Optimization for Dummies by Peter Kent is packed with optimization and marketing gems. This book has excellent reviews and should be on the shelf of everyone who works with web sites
  • 101 Ways to Boost Your Web Traffic: Internet Promotion Made Easier by Thomas Wong is an excellent resource for people who are new to internet marketing. More experienced and knowledgable people may not be happy with this selection but it is good for the absolute beginner.
  • Note that what works and what doesn’t changes so quickly in the world of search engine submissions that recommendations could be out of date. Please refer to the web site links above for the best and most current resources.


Quick Tip
Get Involved:

Do not rely entirely on automatic search engine submission programs. Many of the more popular search engines are starting to require submissions to be entered manually. They stop automatic submissions by displaying a graphic image that contains a code that must be entered with each submission.

Quick Tip

Do not expect immediate results from submissions. Some of the most popular search engines can take up to 8 weeks for submissions to list. Manual review sites (such as DMOZ) may take longer.

Another DMOZ related Tip:
Do not resubmit to the Open Directory Project while waiting for your site to be reviewed. Resubmitting puts your submission back to the end of what is often a very long list of sites. If you need to know the status of your submission you can ask (after 6 weeks) at DMOZ editors moderate those forums. Be sure to read the forum instructions.

What is “SEO”?
SEO stands for Search Engine Optimization. It normally used when referring to contractors that specialize in search engine optimization (Search Engine Optimizer).

Search Engine Optimization doesn’t stop when your site is submitted. It requires an ongoing effort to be sure you maintain good placement.


About Pay Per Click   >