World Wide Web Robots, Wanderers, and Spiders
=============================================

At the rate the Web is currently growing, it would take forever to
manually research all the available Web sites; why not use an
automatic program to do your research and indexing for you?
Robots, wanderers, and spiders are all names for programs that
traverse the Web automatically. This page offers useful
information about these programs, including how to compile and
configure them for use on your Web server. You can also learn
about the problems associated with these programs and how to avoid
technical and diplomatic problems while they run on their own.
This site is your one-stop source of information on everything you
need to know for developing and maintaining your own automatic Web
robot.

Address : http://info.webcrawler.com/mak/projects/robots/robots.html
Year    : 1996
Source  : Microsoft Bookshelf Internet Directory 1996-97
          lycos9510
Tags    : Internet, Tools and Creating Content, Bookshelf 96

------------------------------------------------------------------
gopher://hp84.com/    |    https://ge97.com/
.