Lucene and index updation

[Pradeep bhatt]

How does index gets updated when the content of file changes.

[Lester Burnham]

Not automatically. The indexing code must explicitly delete the old index entry and re-index the file.

[Pradeep bhatt]

ok. that would mean we may miss some files because the index has not been updated.

[Lester Burnham]

Correct. The indexing code needs to strike a balance between freshness of the index and performance when deciding how often to check for updated files/content.

[Aneesh Vijendran]

Yes you need to Commit after you update/delete terms/docs from the Index.

Though these are deleted it still are present in the Index (Marked for deletion), and doesn't come up in ranking/search etc.

[Gian Franco]

Pradeep bhatt wrote:How does index gets updated when the content of file changes.

...it all depends on the tools used to manage the files.

If all your tools are aware of the Lucene index underneath
and they all take care of keeping the index up to date then
you are fine: each tool could funnel the changes on the files
through Lucene's API...

If the files can be managed by a tool that does not update
the index, then you are likely to have/need a batch process
that goes over the files periodically and makes sure that
the index is up to date.

Cheers,

Gian

[Michael McCandless]

One of the nice recent additions to Lucene is a feature called near-real-time search (it's covered in the book -- the IndexWriter.getReader method).

This make the turnaround time between making changes (adds/deletes/updates) to the index, and opening a new searcher that can see these changes, much faster, because you no longer have to .commit or .close the IndexWriter in order to see the changes.

[Pradeep bhatt]

Michael McCandless wrote:One of the nice recent additions to Lucene is a feature called near-real-time search (it's covered in the book -- the IndexWriter.getReader method).

This make the turnaround time between making changes (adds/deletes/updates) to the index, and opening a new searcher that can see these changes, much faster, because you no longer have to .commit or .close the IndexWriter in order to see the changes.

Thanks author for sharing this piece of useful info. From which version has this been introduced ?

[Michael McCandless]

You're welcome!

Near-real-time search was added in 2.9.

[Pradeep bhatt]

Thanks for the info. Do you suggest backing up the index files for critical applications. I think, if index files get lost ,constructing them would take time if we dealing with huge amounts of data.

[Aneesh Vijendran]

Yes very much. Chances of Index files getting corrupted are not rare. I had this situation once when there were data in the index with some unknown/asci/Unicode characters.

Some 30k docs in the Document Manager when indexed, the index size was 7-8 GB.

Imagine if this gets corrupted ;)

Cheers
Aneesh

[David Newton]

Pradeep bhatt wrote:Do you suggest backing up the index files for critical applications.

Using the word "critical" here answers your own question: under what circumstances would you *not* back up "critical" data?!

[Michael McCandless]

The book shows how to take a hot backup of the index (ie backing up in the background even while an IndexWriter is still making changes to the index). The resulting backup is still a point-in-time copy of the index, as of when the backup began.

[Gian Franco]

...is the hot backup a new feature?

I used to have a companion batch program for my
application that, in case of emergencies, would
re-index all the data.

Regarding performance of reindexing large data sets...
Lucene was not the bottleneck...finding the files and
extracting the text took most of the time...

Cheers,

Gian

[Otis Gospodnetic]

Gian Franco wrote:...is the hot backup a new feature?

Yes, it's relatively new.

Otis