Analyzing Visits from Indexing Agents

You can classify specific hosts, user agents, and referrers as robots or spiders. (A spider is a nonhuman agent that visits Web sites in order to index them for its search engine or directory. A robot is a nonhuman agent that visits Web sites for any purpose. For example, a robot could check pages for updates or check to ensure a site is still on the Internet.) This enables you to separate robots and spiders from regular Web site traffic. Robot and spider traffic appears separately in the Robot/Spider Summary and is not counted in the normal visit statistics but it is counted in the number of hits. By default, NetTracker is configured with the most common robots and spiders.

To classify a host, user agent, or referrer as a robot or spider:
  1. Click the Options tab.
  2. In the left-hand panel, click Robots/Spiders under Marketing.
  3. Click Add.
  4. In the Name of spider box, enter the name you want to have displayed for the robot/spider group.
  5. Click Save Options and then click Continue.

Adding Visitors to a Robot/Spider Group

If you are adding visitors to the robot/spider group immediately after creating it, start with step 3. Otherwise, start with step 1.

To add visitors to a robot/spider group:
  1. Click the Options tab.
  2. In the left-hand panel, click Robots/Spiders under Marketing.
  3. Select the robot/spider group and click Visitors.
  4. Click Add.
  5. Enter a filter that describes the host, user agent, or referrer you want to classify as a robot or spider, and click Add. (For details on filters, see "Filtering a NetTracker Option".)
  6. The statistics for this host, user agent, or referrer will be classified as a robot or spider in all new log file data you import from now on. If this is what you want or if you have additional hosts, user agents, or referrers to enter at this time, click Continue. If you want to classify this host, user agent, or referrer as a robot or spider in data you have already imported, you will need to delete all the data in your profile and reimport your log files. (Note: If you do not still have all the log files you have imported into this profile, you will lose data permanently by choosing this option.) To delete your profile data, click Delete Profile Data.

Editing the Robot/Spider Group

You can make changes to the robot/spider groups at the list of groups.

To
Select the group in the list and
Change the name
Click Edit
Delete a group
Click Delete
Change the visitors in the group
Click Visitors

Editing Visitors in the Robot/Spider Group

You can also make changes to the hosts, user agents, and referrers in the robot/spider group at the list of visitors for the group.

To
Select the visitor and
Delete a host, user agent, or referrer
Click Delete
Edit the host, user agent, or referrer
Click Edit

Determining the Group for Visitors that Fit Multiple Groups

A host, user agent, or referrer can only belong to one robot/spider group. If you have a visitor that meets the criteria for more than one group, NetTracker will include the visitor in the first robot/spider group it can belong to in the list of robot/spider groups. You can use the Move Up and Move Down buttons to change the order of the robot/spider groups in the list.