• Post Reply Bookmark Topic Watch Topic
  • New Topic

How to configure ProxyMesh in Java  RSS feed

 
Ramakrishna Udupa
Ranch Hand
Posts: 254
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi All,

I want to crawl some websites. If I crawl more, They will block the IP address. For that I think proxy-Mesh is the better one. How to crawl the web-site using Proxy-Mesh. Is there any better solutions to change IP address through java continuously or any other idea for this?

Thanks:
Ramakrishna K.C
 
Jelle Klap
Bartender
Posts: 1952
7
Eclipse IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
If a website doesn't want to be crawled, you shouldn't crawl it. Period.
Always be responsible when building a crawler and look for a site's robots.txt and robot meta tags. Adhere to the permissions specified by those.
 
Ramakrishna Udupa
Ranch Hand
Posts: 254
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Nope, The site can be crawled. But limited number of times. So I'm asking. I want to crawl that site by using proxy's. Is there any better solutions?
 
Don't get me started about those stupid light bulbs.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!