• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Any Open Source Search Engine with Auto Crawling?

 
asr chowdary
Ranch Hand
Posts: 35
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi

We have Lucene Search Engine(Open Source),but it dosen't have Crawling. Any Other Search Engine with Crawling??


Thanks
 
M Shareef
Greenhorn
Posts: 6
Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
may be this link helpful for you:

http://java-source.net/open-source/crawlers

 
M Shareef
Greenhorn
Posts: 6
Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
You can go to Solr serach engine which is build on Lucene.

http://lucene.apache.org/solr/
 
Hussein Baghdadi
clojure forum advocate
Bartender
Posts: 3479
Clojure Mac Objective C
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
M Shareef wrote:You can go to Solr serach engine which is build on Lucene.

http://lucene.apache.org/solr/


Lucene isn't a search engine, Lucene is an IR (Information Retrieval) library. Also Solr doesn't provide any crawling facilities.
To have crawling, you might want to use "Apache Nutch".
 
Luan Cestari
Ranch Hand
Posts: 172
C++ Redhat Ruby
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I used Nutch some time ago. It is a very nice project and you can easily fit different proposes (I know that some big companies use it). Bixo (openbixo) seems to be very nice (I didn't tested yet). Depending your propose and your time I would say to create your own using some parallel programming (there is a lot of details in this part, like using a bloom filter to store the URL already fetched ) and a database (cassandra, e.g.) to store.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic