• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Liutauras Vilda
  • Jeanne Boyarsky
  • Devaka Cooray
  • Paul Clapham
Sheriffs:
  • Tim Cooke
  • Knute Snortum
  • Bear Bibeault
Saloon Keepers:
  • Ron McLeod
  • Tim Moores
  • Stephan van Hulst
  • Piet Souris
  • Ganesh Patekar
Bartenders:
  • Frits Walraven
  • Carey Brown
  • Tim Holloway

most recent 3 - 4 posts not included in Lucene reindex

 
Ranch Hand
Posts: 17424
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
In our mysql database, we only have about 15 posts. Post ids 14 - 34, with a few posts missing in the middle. When I start a new reindex from the admin control panel, the newest 3 - 4 are not included in the new index. Also, during debugging, I ran into a few file locking problems caused by handles not being freed. My code fixes are posted below. I built the project from source. I pulled from cvs on 05 Feb 2008.

This fixed the indexing problem... (LuceneIndexer.java)

private void createRAMWriter() {
try {
if (this.ramWriter != null) {
this.ramWriter.flush();
this.ramWriter.close();
}
this.ramDirectory = new RAMDirectory();
this.ramWriter = new IndexWriter(this.ramDirectory, this.settings.analyzer(), true);
this.ramNumDocs = SystemGlobals.getIntValue(ConfigKeys.LUCENE_INDEXER_RAM_NUMDOCS);
} catch (IOException e) {
e.printStackTrace();
throw new SearchException(e);
}
}

These fixes helped the file locking problem... (LuceneIndexer.java)

public void flushRAMDirectory() {
synchronized (MUTEX) {
IndexWriter writer = null;
boolean success = true;
try {
writer = new IndexWriter(this.settings.directory(), this.settings.analyzer());
writer.addIndexes(new Directory[] { this.ramDirectory });
writer.optimize();

this.createRAMWriter();
} catch (Exception e) {
success = false;
e.printStackTrace();
throw new SearchException(e);
} finally {
if (writer != null) {
try {
writer.flush();

} catch (Exception e) {
e.printStackTrace();
}
try {
writer.close();

} catch (Exception e) {
e.printStackTrace();
}
}
if (success)
this.notifyNewDocumentAdded();

}
}
}
[originally posted on jforum.net by bwfrieds]
 
Migrated From Jforum.net
Ranch Hand
Posts: 17424
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
bug # JF-831
[originally posted on jforum.net by bwfrieds]
 
Migrated From Jforum.net
Ranch Hand
Posts: 17424
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
After a deployment to IBM Websphere Portal on linux, I saw the problem was not yet fixed. More changes were necessary to get the reindexing to behave reliably. Those changes are below. I placed my initals next to my changes. I think the final fix was the call for writer.flush() before the call to optimize() and after the call to addDocument().

Occasionally I will still see an IOException during testing; when trying to update the .cfs files. I don't know where that file handle is not being released.

LuceneIndexer.java

public void batchCreate(Post post) {
synchronized (MUTEX) {
try {
Document document = this.createDocument(post);
this.ramWriter.addDocument(document);
this.ramWriter.flush(); // bwf
this.flushRAMDirectoryIfNecessary();
} catch (IOException e) {
e.printStackTrace(); //bwf
throw new SearchException(e);
}
}
}

private void createRAMWriter() {
try {
if (this.ramWriter != null) {
this.ramWriter.flush();// bwf
this.ramWriter.close();
}
this.ramDirectory = new RAMDirectory();
this.ramWriter = new IndexWriter(this.ramDirectory, this.settings.analyzer(), true);
this.ramNumDocs = SystemGlobals.getIntValue(ConfigKeys.LUCENE_INDEXER_RAM_NUMDOCS);
} catch (IOException e) {
e.printStackTrace();// bwf
throw new SearchException(e);
}
}

public void flushRAMDirectory() {
synchronized (MUTEX) {
IndexWriter writer = null;
boolean success = true;
try {
writer = new IndexWriter(this.settings.directory(), this.settings.analyzer());
writer.addIndexes(new Directory[] { this.ramDirectory });
writer.flush(); // bwf
writer.optimize();

this.createRAMWriter();
} catch (Exception e) {
success = false; //bwf
e.printStackTrace();//bwf
throw new SearchException(e);
} finally {
if (writer != null) {
try {
writer.flush();

} catch (Exception e) {
e.printStackTrace();
}
try {
writer.close();

} catch (Exception e) {
e.printStackTrace();
}
}
if (success)
this.notifyNewDocumentAdded();
}
}
}

private void optimize(IndexWriter writer) throws Exception {
if (writer.docCount() % 100 == 0) {
if (logger.isInfoEnabled()) {
logger.info("Optimizing indexes. Current number of documents is " + writer.docCount());
}
writer.flush(); // bwf
writer.optimize();

if (logger.isDebugEnabled()) {
logger.debug("Indexes optimized");
}
}
}

[originally posted on jforum.net by bwfrieds]
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!