The World Wide Web contains more than ten million websites with thousands more being added daily from all over the world, and search engines are tasked with presenting the most relevant pages based on the search criteria entered. Finding a large and popular site like irs.gov is easy because it is related to a large agency, but a small site like mine consumermortgagereduction.com would be impossible without the use of search engines. The task of finding a website that relates to a specific subject would be almost without the use of the search engine, yet they perform this task with relative ease using tools that may be unknown to most users of search engines.

The first of these tools utilize by search engines are web-crawlers; they search sites based on meta-tags, or the hidden descriptions of the site encoded in the head of the web page. By using web-crawlers to spider websites, the search engine does not have to search the entire webpage to return results, just the first few lines of code. The majority of the code web-crawlers spider is contained in the head section or top of the web page, and search engines use this information to list relevant search terms results.

Meta-bots and Google-bots are used by search engines in the same way that web-crawlers are deployed, and the only difference is that bots read the first lines of text that is visible on the browser of the user. Meta-bots and Google-bots are useful in providing content oriented sites with a quick description of the site, and is use to match the relative words in a search engine query.

The indexer is not a search engine par say, for example the largest indexer is Yahoo, which is not a search engine like Google that is a true search engine. A true search engine uses keyword searches to find website, whereas an indexer stores a link to sites based on categories. To submit a website for inclusion into an indexer the submitter has to find a relevant category that matches their website, but to submit to a search engine they only have to submit the site Universal Resource Listing or URL and the search engine deploys its web-crawler or bots to index the site.

The search engine has become an intricate part of life, especially in the lives of students that utilize the web for research. Without the search engine the search for sites that contain the terms being searched for would almost be impossible to locate in a world of millions of websites.