Slurp

Slurp is the spider for the Inktomi search engine database. Inktomi is not a search engine - rather, it is a database which is sold either wholly or in part to other search engines. Search engines which use the Inktomi database include Hotbot and Snap. Until recently Yahoo used the Inktomi database, but they switched to Google.

Inktomi is pronounced INK TUH ME and means "crafty Indian Spider". These days Inktomi is generally used to backup the results from a directory listing. This search database is becoming less and less used as they are simply too expensive when compared to other alternatives.

Slurp obeys the contents of your Robots.Txt file as well as the Robots Metatag. Like most search engine spiders, Slurp follows HREF tags. It does not follow SRC tags.

In general, Slurp will not index documents which are dynamically created. They recommend that you create static copies of dynamic pages so that Slurp will index them.

Slurp will reindex a site once every three to four weeks.