Forums Register Login

Crawler

+Pie Number of slices to send: Send
Hi,

How to know if it is a web crawler??

Thanks in advance.
Sonal.
+Pie Number of slices to send: Send
Sonal,
A web crawler is a program that goes through web sites looking for links and then goes to those links and so on.
+Pie Number of slices to send: Send
Are you asking "How can you tell if the client requesting a page is a crawler or a browser?" I don't think you can. A crawler could be written to send headers that perfectly impersonate a given browser. I wonder if you could build the smarts to notice a particular session running up a lot of bandwidth in a short time.
What are you doing? You are supposed to be reading this tiny ad!
a bit of art, as a gift, that will fit in a stocking
https://gardener-gift.com


reply
reply
This thread has been viewed 775 times.
Similar Threads
Code for recursively downloading pages from particular web site(url)?
A Question on Lucene
<terminated, exit value: 0>C:\Program Files\Java\jre6\bin\javaw.exe (Feb 12, 2009 1:54:50 PM)
jdbc - common problems/mistakes
Art of Java
More...

All times above are in ranch (not your local) time.
The current ranch time is
Mar 29, 2024 07:05:58.