Scooter

Scooter is the spider for AltaVista. What this spider does is scan websites all day long, looking for things to add to the AltaVista index. Thousands of threads are sent out simultaneously all day, each and every day, to all of the corners of the world wide web.

This spider spends it's time scanning web pages for hyperlinks and text to add to the index. Each page is torn apart and a specific algorithm is applied to determine how (and if) all of the information is to be added to the monstrous index.

Scooter visits pages which are submitted to AltaVista via the "Add URL" link. In addition, it revisits old pages to determine if there were any changes which need to be updated into the index. Pages which no longer exists are deleted if they continue to get errors on several visits (this implies that 404 errors will always exist as they are not deleted right away). And of course, new links that are found are investigated by Scooter - sometimes.