Learn about Crawlers and Directories.

Directories are human powered search engines that do not crawl the Web while crawler based search engines like Google create their individual listings automatically. They “crawl” or “spider” websites on the Internet and assist the browsers to search through the available data bank / information. The crawler based search engines can also locate the alterations in the web pages. During early times search engines displayed crawler based results or human powered listings. However, nowadays both types are displayed where the hybrid search engine favors one type of listing over the other depending on search terms, queries and information required by web users.

Directories, on the other hand, are human powered, like Open Directory, is entirely dependent on humans for its web content listing. The client is asked to submit to the Directory a descriptive account of their website. In directories, repeated changes in web content do not influence the page listing of the website.


The crawler based search engines function under the three major elements: firstly, the crawler, also called spider, studies the web content and follows the links to explore other pages within the website. The term “crawled” or “spidered” usually refers to this operation. The crawling of the site is a continuous process carried out almost every month that helps to locate changes incorporated in the website. After the crawling process is over, the information gathered is indexed, the second element of the search engine. This is referred to either as an index or catalogue and is an enormous book that retains all the details of every web page collected by the “spider”. In the event of any changes being implemented in the web page, the book is updated with the current information. At times the new pages with alterations may not be indexed, though it has been “spidered”, and unless the web page is indexed it is not available for those searching information at websites in the search engine.


The search engine software is another element of crawler based search engines. This system or programming serves millions of pages recorded in the index to find matches to a search and ranks them according to their relevance. Almost all crawler based search engines have the above mentioned features but there are differences within the functional aspects. This often leads to differences in results while searching for the same search term.


The Open Directory in the World Wide Web is also known as “link directory” as it specializes in linking one website to other websites and also categorizes the links. As a Directory is not a search engine, it does not display the web page listing based on keywords rather it lists the websites on the basis of their categories and sub categories. The categorizing process depends on the entire website instead of being based on one web page or a set of keywords. Web directories allow the website owners to directly submit their site for inclusion and have editors to review their sites for judging the relevance of their contents. These directories are normally general in scope but lists websites covering a wide range of categories, regions and languages. The niche directories, however, focus on single language and specialist sectors for selected regions. The general web directories are Yahoo, Open Directory and other such.


The various features included in directories are :

  • Free submission
  • Reciprocal link
  • Paid submission
  • Featured listing
  • Bid for  position
  • Affiliate links.


The world’s largest website directory has been primarily built from quality free submissions. The cross referencing facilities through link ensure that a client’s website, is added to the state and world directory automatically.

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *