Search doesn't work on Chinese keywords

[Migrated From Jforum.net]

Dear Rafael,
I got JForum 2.0.2 up and running on my Tomcat 5.5.4+JDK1.5.0+PostgreSQL 7.4/RedHat Linux 9 box. I use UTF-8 encoing. The Chinese input is fine. But I can't search back the post. Is there anything wrong about the search? There is always 0 result.

BTW. I got a JavaScript warning on the search.htm page. I think it's better to add a name="formSearch" at the end of the <form> tag
[originally posted on jforum.net by andowson]

[Migrated From Jforum.net]

Hi,
I find a way to fix this problem. Just modify the insertSearchWords() method in net/jforum/drivers/generic/SearchModel.java. Find the following code fragment.

   // Skip words less than 3 chars			
   if (words[i].length() < 3) {
       continue;
   }

change it into the following:

// Skip words less than 1 chars			
   if (words[i].length() < 1) {
       continue;
   }

It will generate every word length longer than 1 into a keyword.

BTW. There is a little bug on searching capital word. JForum generates keyword in lowercase, for example, ok. However, if you input a keyword in uppercase, for example, OK. You won't get any result. To fix this bug, just modify the topicsByKeyword() method in this file again. Find the following code fragment.

// Get the post ids to which the words are associated to
    for (int i = 0; i < sd.getKeywords().length; i++) {
        p.setString(1, "%" + sd.getKeywords()[i] + "%");

Change it into the following:

// Get the post ids to which the words are associated to
    for (int i = 0; i < sd.getKeywords().length; i++) {
        p.setString(1, "%" + sd.getKeywords()[i].toLowerCase() + "%");

recompile it and put the generate SearchModel.class to replace the original one.
[originally posted on jforum.net by andowson]

[Migrated From Jforum.net]

For large forums it could be useful to integrate a text search engine like lucene.


http://jakarta.apache.org/lucene/docs/index.html
[originally posted on jforum.net by marc]

[Migrated From Jforum.net]

OK, I changed the code as you said.

However, inserting every word can make the index a little messy, as well I removed the 'LIKE' sintax since it does not perform well on large forums.

Probably I'll put some configuration option to these issues.. something like "search.index.all.words" and "seach.use.like".

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Lucene is in mind for a long time, but integration is not that easy. Implementations are welcome.

Rafael
[originally posted on jforum.net by Rafael Steil]

[Migrated From Jforum.net]

Yes, lucene is a waiting feature. I see mvnForum hava use it. But they need to set URLEncoding=UTF-8 in Tomcat 5's Connector setting for Chinese fix.
If you've developed it in JForum, I'll help to test it in Chinese environment.

Before Lucene is integrated, the search still needs to work. So, I use this every word is keyword strategy. The users will think this forum is useful when they really post an article and they can search it back later. If they can't search back, they will think this forum is wrong no matter you use what kind of search engine.

Andowson
[originally posted on jforum.net by Anonymous]

[Migrated From Jforum.net]

Well, I can add these config options I mentioned. It will be great if you could help with Chinese ( and therefore all cyrillic languages )!

Rafael
[originally posted on jforum.net by Rafael Steil]