Firebot is a discrete spider(bot) program which help you index/fetch the whole Internet.
A Multi-threads Spider(Web Crawler) Written in C