Define “Spiders” – How do “Spiders” Work.

Directories are human powered search engines while crawler based search engines like Google create individual listings automatically. They “crawl” or “spider” websites on the Internet and assist the browsers to search through the available data bank / information. The crawler based search engines can also locate the alterations in the web pages. During early times search engines displayed crawler based results or human powered listings. However, nowadays both types are displayed where the hybrid search engine favors one type of listing over the other depending on search terms, queries and information required by web users.


The crawler based search engines function under the three major elements: firstly, the crawler, also called spider, studies the web content and follows the links to explore other pages within the website. The term “crawled” or “spidered” usually refers to this operation. The crawling of the site is a continuous process carried out almost every month that helps to locate changes incorporated in the website. After the crawling process is over, the information gathered is indexed, the second element of the search engine. This is referred to either as an index or catalogue and is an enormous book that retains all the details of every web page collected by the “spider”. In the event of any changes being implemented in the web page, the book is updated with the current information. At times the new pages with alterations may not be indexed, though it has been “spidered”, and unless the web page is indexed it is not available for those searching information at websites in the search engine.


The search engine software is another element of crawler based search engines. This system or programming serves millions of pages recorded in the index to find matches to a search and ranks them according to their relevance. Almost all crawler based search engines have the above mentioned features but there are differences within the functional aspects. This often leads to differences in results while searching for the same search term. After the entries are annotated or classified under various topics, they are subsequently ranked by Google’s page rank algorithm. Search engine submission is the manner of submitting a website directly by a Webmaster to a search engine. This method is often seen as a way to promote a website but generally it is not necessary since the major search engines like Google, Yahoo and MSN use crawlers, bots, and spiders that would eventually identify by themselves most web sites on the Internet. Primarily

there are two basic reasons to submit a website or web page to search engines, firstly, to add an entirely new web site as the site operators do not  want to wait for a search engine to discover them. Secondly they would like the web page or web site to be updated in the respective search engines.


This web directory blends Google’s hi-tech search technological strategies with Open Directory pages and develops one of the most authentic and significant tools for exploring information on the web. Though websites included in the directory are lesser in number but normally these are of high quality. With Google Directory, the web user can search or browse pages of interest. It combines human edited directory and a “spider” based index.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *