Spider

A program that downloads web pages and follows links to explore every web page it can find, usually with the goal of indexing every web page for a search engine, but also used to do other forms of research and analysis on web structures, such as linguistic research studying the use of language online.