Are you asking "How can you tell if the client requesting a page is a crawler or a browser?" I don't think you can. A crawler could be written to send headers that perfectly impersonate a given browser. I wonder if you could build the smarts to notice a particular session running up a lot of bandwidth in a short time.
Post by:autobot
What are you doing? You are supposed to be reading this tiny ad!
a bit of art, as a gift, that will fit in a stocking