It depends on what you are searching.. So far as writing of search engine is concern you actually need to start with on URL and start indexing those pages..ALso while indexing those pages you take note of the different URL you encounter and loop thru them.. But
you should be very careful while doing that else might end up in mess.
Pratically Search engine like google indexes the net and uses parallel comp to execute your search criteria.
------------------