Basics

A search engine works much like a database. Information about Web pages is gathered and stored. When a user types in a search phrase, this acts like a query, sending the engine searching through the information it has already collected and pulling out the pages in a directory format, the most relevant pages first.

Spiders

Search engines use technology known as spiders to find a Web page and pass relevant information to indexing software. This information is then stored in the engine's database and accessed when a user types in a specific search term.

How Spiders Find New Pages

The spiders get links to new Web pages by following links listed on pages it has already searched, much like a treasure hunt. Spiders do not intuitively know about new pages added to the Internet unless the author registers the page with the search engine or the spiders pick up links to that page from other pages.

Indexing

Indexing programs grab search terms, text, images and links from a Web page and catalogs it in the search engine's databases.

Considerations

Many pages are excluded from a search engine because of content restrictions. These pages may or may not show up in one search engine results. It is recommended that when searching for information on the Web that you use multiple search engines to see as many pages as possible on the subject because what is excluded from one engine might not be on another.

Source:

Berkley University