Need Some Help with Lucene

[Harish Tam]

Hello All,


I have been doing some RnD on Lucene search API and found out that it is best suited for indexing HTML files, I would like to know if there is any way to index jsp files,
I have tried to use spindle which is a indexer which uses lucence inside it, Also I have a web application which has jsp as well as html files inside it, but lucene is not able to index the html files within the web app, can anyone help me with this query��.Also I tried to index www.sun.com web site and it did not get index at all where are the rest of the jsp based site such www.jspin.com were indexed. Is there any specific setting with the web server to restrict this or something else?

Regards
Harish

[Ulf Dittmer]

I don't think it is a good idea to try to index public web sites. Google already does a very good job at this. You would need to think about how you retrieve the pages, follow the links inside of the pages, deal with JavaScript that may be changing the links etc. Lucene does not deal with any of these (maybe the spindle thing you mention does).

A collection of HTML (or JSP) pages on your local hard disk would be a different matter.
[ January 20, 2006: Message edited by: Ulf Dittmer ]

[Harish Tam]

OK I agree,

But, what is i have created a web application adn hosted it on my local server, say http://one.server.com, and this application does have jsp and html pages included within it, i am not even able to index this pages, i have tried the Demo3 example that is given in the Lucene demo kits, and it is perfectly working with HTML files....

But now with Jsp pages :-(

[Ulf Dittmer]

In what way is it not working? Do you treat JSP pages differently from HTML pages? Why don't you post a short code excerpt that shows how you access HTML and JSP pages, so that we can take a look at it.