most recent 3 - 4 posts not included in Lucene reindex

[Migrated From Jforum.net]

In our mysql database, we only have about 15 posts. Post ids 14 - 34, with a few posts missing in the middle. When I start a new reindex from the admin control panel, the newest 3 - 4 are not included in the new index. Also, during debugging, I ran into a few file locking problems caused by handles not being freed. My code fixes are posted below. I built the project from source. I pulled from cvs on 05 Feb 2008.

This fixed the indexing problem... (LuceneIndexer.java)

private void createRAMWriter() {
try {
if (this.ramWriter != null) {
this.ramWriter.flush();
this.ramWriter.close();
}
this.ramDirectory = new RAMDirectory();
this.ramWriter = new IndexWriter(this.ramDirectory, this.settings.analyzer(), true);
this.ramNumDocs = SystemGlobals.getIntValue(ConfigKeys.LUCENE_INDEXER_RAM_NUMDOCS);
} catch (IOException e) {
e.printStackTrace();
throw new SearchException(e);
}
}

These fixes helped the file locking problem... (LuceneIndexer.java)

public void flushRAMDirectory() {
synchronized (MUTEX) {
IndexWriter writer = null;
boolean success = true;
try {
writer = new IndexWriter(this.settings.directory(), this.settings.analyzer());
writer.addIndexes(new Directory[] { this.ramDirectory });
writer.optimize();

this.createRAMWriter();
} catch (Exception e) {
success = false;
e.printStackTrace();
throw new SearchException(e);
} finally {
if (writer != null) {
try {
writer.flush();

} catch (Exception e) {
e.printStackTrace();
}
try {
writer.close();

} catch (Exception e) {
e.printStackTrace();
}
}
if (success)
this.notifyNewDocumentAdded();
}
}
}
[originally posted on jforum.net by bwfrieds]

[Migrated From Jforum.net]

bug # JF-831
[originally posted on jforum.net by bwfrieds]

[Migrated From Jforum.net]

After a deployment to IBM Websphere Portal on linux, I saw the problem was not yet fixed. More changes were necessary to get the reindexing to behave reliably. Those changes are below. I placed my initals next to my changes. I think the final fix was the call for writer.flush() before the call to optimize() and after the call to addDocument().

Occasionally I will still see an IOException during testing; when trying to update the .cfs files. I don't know where that file handle is not being released.

LuceneIndexer.java

public void batchCreate(Post post) {
synchronized (MUTEX) {
try {
Document document = this.createDocument(post);
this.ramWriter.addDocument(document);
this.ramWriter.flush(); // bwf
this.flushRAMDirectoryIfNecessary();
} catch (IOException e) {
e.printStackTrace(); //bwf
throw new SearchException(e);
}
}
}

private void createRAMWriter() {
try {
if (this.ramWriter != null) {
this.ramWriter.flush();// bwf
this.ramWriter.close();
}
this.ramDirectory = new RAMDirectory();
this.ramWriter = new IndexWriter(this.ramDirectory, this.settings.analyzer(), true);
this.ramNumDocs = SystemGlobals.getIntValue(ConfigKeys.LUCENE_INDEXER_RAM_NUMDOCS);
} catch (IOException e) {
e.printStackTrace();// bwf
throw new SearchException(e);
}
}

public void flushRAMDirectory() {
synchronized (MUTEX) {
IndexWriter writer = null;
boolean success = true;
try {
writer = new IndexWriter(this.settings.directory(), this.settings.analyzer());
writer.addIndexes(new Directory[] { this.ramDirectory });
writer.flush(); // bwf
writer.optimize();

this.createRAMWriter();
} catch (Exception e) {
success = false; //bwf
e.printStackTrace();//bwf
throw new SearchException(e);
} finally {
if (writer != null) {
try {
writer.flush();

} catch (Exception e) {
e.printStackTrace();
}
try {
writer.close();

} catch (Exception e) {
e.printStackTrace();
}
}
if (success)
this.notifyNewDocumentAdded();
}
}
}

private void optimize(IndexWriter writer) throws Exception {
if (writer.docCount() % 100 == 0) {
if (logger.isInfoEnabled()) {
logger.info("Optimizing indexes. Current number of documents is " + writer.docCount());
}
writer.flush(); // bwf
writer.optimize();

if (logger.isDebugEnabled()) {
logger.debug("Indexes optimized");
}
}
}

[originally posted on jforum.net by bwfrieds]