too many open files with lucene

[rr chauhan]

I am using lucene for indexing the files and then these indexes are used to search/update or delete records in files.Apart from that whenever a new record is created in database it also triggers the creation of the record in lucene based file index system.
Compass core APi is used for interaction with lucene.
This worked fine in development and QA environments but failed in production with the error that "Too many files open".
This caused our server to hang.
These are the details of the environment that we are working with.
Server Tomcat 5.5.16
OS Linux
Lucene core API version 2.2
Compass core API revision 1.2.1
The compass core parameters used for lucene are
merge factor 8000 (this was changed to 200 later on but the error still existed)
lock timeout 200 secs
optimizer.schedule was set to true
schedule.period to 200 secs
and commitBeforeCompletion is set to true

As a part of initial anlysis we found out that at the time when this error occured the number of open files on the Linux box were 31k which is fairly large number. I feel the problem has to be with lucene rather than other applications deployed on the box coz once we stopped the application that was using lucene evrything worked out smoothly.

If anyone has ever encountered any such issues please let me know how you debugged the actual issue and the steps you took to resolve them.

Rgds
Ramraj

[Joe Ess]

Originally posted by rr chauhan:
I feel the problem has to be with lucene

I would hazard a guess that the problem isn't lucene, it's with the application that's using lucene. The Lucene FAQ has some advice for solving this error.
I'd add that lsof can tell you what files are open to help you find the leak.

[rr chauhan]

Hi Joe
I have gone through the lucene faqs and found nothing amiss in out application. I feel its the optimizer parameters that are not behaving correctly. It may not be merging the index files at required intervals.

Rgds
Ramraj

[Ulf Dittmer]

Did you follow Joe's advice to check which files are actually open? Is there an excessive number of Lucene index files?

[rr chauhan]

after debugging for a while it was found that optmizer thread is not running.The following properties were provided for the compass indexer in the spring config file.
<prop key="compass.engine.optimizer.schedule">true</prop>
<prop key="compass.engine.optimizer.schedule.period">200</prop>
<prop key="compass.engine.optimizer.schedule.daemon">true</prop>
<prop key="compass.engine.optimizer.type">
org.compass.core.lucene.engine.optimizer.AdaptiveOptimizer</prop>
<prop key="compass.engine.optimizer.adaptive.mergeFactor">200</prop>

The Optimizer doesn't pick up these properties
corresponding to
compass.engine.optimizer.schedule.daemon
compass.engine.optimizer.schedule
and
compass.engine.optimizer.schedule.period
and doesn't scheule a daemon thread for Optimizer. Other properties are picked up fine.
Is it imperative to start the daemon thread manullly when loading the compassIndexer programmtically?Is there any way we could just have the configuration changes and have the optimizer thread running?
These properties have been defined in the spring configuration file which is used to initialize the Compass Indexer bean.
--Rgds
Ramraj