Search engines use web “spiders” known as “Robots” to create the index for their search engines. These robots load pages and follow hyper links to report the text. Robots .txt file is a file created by “spiders” to find information and methods of cataloguing a website. This is an ASCII text file located at the document root of the server. Normally, it defines and catalogues the documents / directories that reputed search engine “spider” are forbidden to index. The conception of Robot Exclusion Protocol was visualized by Martin Koster in 1994 to deal with the problems that had arisen with the increasing use of the Internet. Some of the consistent problems caused by the rapid firing requests by robots involved loading pages in quick succession. Other common problems faced were robots indexing the information from HTML directory trees, temporary information cataloguing and even accessing cgi-scripts. This resulted in Robot Exclusion Protocol being readily accepted by Webmasters and web robot makers in order to control and organize the indexing procedure.
The growth and popularity of Internet is phenomenal with millions of people using the system. Today the number of web robots crawling the sites has increased tremendously and it has become exceedingly important for site owners to create and maintain the robots .txt file. The Robot Generator enables the site owner to create exclusive robot .txt files by selecting all robots or any specific user agent. This can be followed by incorporating all documents and directives by inserting the path name manually or by the usage of FTP. Once the restriction and directives have been established, the robot .txt files can be saved in the hard drive or can be directly uploaded to the server.
It has, however, to be maintained that robot exclusive files are not solely meant for security reasons. Few robots tend to ignore the file while others purposely load the documents / files that have been marked “disallowed”. However, such robot exclusion files can be used extensively for controlling the file / document visibility in search engines. Robot Generator allows web users to create robot .txt files quickly that are needed for instructing search engines about those segments which would not be indexed and denied access to the general web public. Website owners use the robot .txt files to provide instruction about their site to web robots called “The Robots Exclusion Protocol”. There are two important guidelines to be considered when using robot .txt file :
01) Certain robots ignore a website robot .txt like Malware Robots which scan the web for security vulnerabilities. Even the e-mail address harvesters used by spammers do not heed the robot .txt file.
02) Robot .txt file is available for all web users. Any web client can ascertain the sections of the server which they do not want the robots to access. In such cases, it is prudent not to use robots .txt for hiding information.
Robot .txt is a de facto standard and is not a privately owned body. They are generally inserted in the top level directory of the web server. For website owners it is imperative to put the robot .txt files in its proper place on a web server so that the resulting URL can function accurately. The robot .txt file is a text file with a single or multiple records.