Lucene - get TermVector positions

[Allasso Travesser]

Hello,

re. Lucene:

I am indexing text using Field.TermVector.WITH_POSITIONS in order to use it for highlighting and other post processing. I have not been able to find out how to access this information during the search.

Can anyone give me a pointer?

Thanks,

Allasso

[Karthik Shiraly]

IndexReader.getTermFreqVector() methods

[Allasso Travesser]

Yes, I have tried that, however, I get a compile time error:

non-static method getTermFreqVector(int,java.lang.String) cannot be referenced from a static context.

being that IndexReader is an abstract class, I can't instantiate it either.

Is there something I am missing here?

thanks for your reply, Allasso

[Karthik Shiraly]

Hi Allasso,

A concrete IndexReader object is constructed the normal way, i.e, using its factory method:

IndexReader reader = IndexReader.open(<your index directory path>);

Note: Ensure that your writers have been close()d, before getting a reader. Lucene has a kind of versioning concept for the index; to use the latest version, all the writers should be closed.

Perphaps IndexReader's termPositions() method may also prove useful to you.

Cheers
Karthik

[Allasso Travesser]

Thank you X 100, Karthik. I was turning grey over that one.

Since IndexReader is an abstract class, I assumed it could not be instantiated. The Sun Java tutorial I and I section reads:

"An abstract class is a class that is declared abstract—it may or may not include abstract methods. Abstract classes cannot be instantiated, but they can be subclassed."

I guess I need to read up on factory methods. Is "reader" in your example considered an object, or is it considered something else? Is using the open() method a way of subclassing IndexReader?

thanks again,

Allasso

[Karthik Shiraly]

Hi Allosso,

'IndexReader.open()' internally creates an object(=instance) of a concrete (i.e., non-abstract) subclass of IndexReader, using new operator.
'reader' is a local variable which is a reference to that object.
'open()' is not subclassing IndexReader; it's just a method to read a Lucene index, and one of its steps is to create an object of a concrete subclass of IndexReader. Such methods which internally choose a particular subclass to instantiate, and return a reference to that instance, are called 'factory methods' - it's a design pattern (DesignPatternFaq)

Cheers
Karthik

[Allasso Travesser]

thank you, Karthik,

sometimes just a few words in the right direction can get one on his way to some productive learning and save a lot of head banging.

I appreciate your thoughtfulness.

Allasso

[Allasso Travesser]

Examples work really well for me, so I like to post the successful fruits for the benefit of others in the future...

This will print both the term positions (eg, the nth word in the original indexed content) and the beginning and ending character offsets of the queried term.

NOTE: This example only works for a single query term, otherwise you need to iterate over the query terms as noted in the "display terms" section below.

  
public class Search {
    
    public static void main(String[] args) throws IOException, ParseException{
  
        Search s = new Search();
        s.doSearch(args[0], args[1]);
    }
    
    Search(){}
    
    public void doSearch(String db, String querystr) throws IOException, ParseException{
        
        // 1. Specify the analyzer for tokenizing text.
        //    The same analyzer should be used as was used for indexing
        StandardAnalyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
    
        Directory index = FSDirectory.open(new File(db));
    
        // 2. query
        Query q = new QueryParser(Version.LUCENE_CURRENT, "contents", analyzer).parse(querystr);
    
        // 3. search
        int hitsPerPage = 10;
        IndexSearcher searcher = new IndexSearcher(index, true);
        IndexReader reader = IndexReader.open(index, true);
        searcher.setDefaultFieldSortScoring(true, false);
        TopScoreDocCollector collector = TopScoreDocCollector.create(hitsPerPage, true);
        searcher.search(q, collector);
        ScoreDoc[] hits = collector.topDocs().scoreDocs;
        
        // 4. display term positions, and term indexes 
        System.out.println("Found " + hits.length + " hits.");
        for(int i=0;i<hits.length;++i) {
            
            int docId = hits[i].doc;
            TermFreqVector tfvector = reader.getTermFreqVector(docId, "contents");
            TermPositionVector tpvector = (TermPositionVector)tfvector;
            // this part works only if there is one term in the query string,
            // otherwise you will have to iterate this section over the query terms.
            int termidx = tfvector.indexOf(querystr);
            int[] termposx = tpvector.getTermPositions(termidx);
            TermVectorOffsetInfo[] tvoffsetinfo = tpvector.getOffsets(termidx);
            
            for (int j=0;j<termposx.length;j++) {
                System.out.println("termpos : "+termposx[j]);
            }
            for (int j=0;j<tvoffsetinfo.length;j++) {
                int offsetStart = tvoffsetinfo[j].getStartOffset();
                int offsetEnd = tvoffsetinfo[j].getEndOffset();
                System.out.println("offsets : "+offsetStart+" "+offsetEnd);
            }
            
            // print some info about where the hit was found...
            Document d = searcher.doc(docId);
            System.out.println((i + 1) + ". " + d.get("path"));
        }
    
        // searcher can only be closed when there
        // is no need to access the documents any more. 
        searcher.close();
    }
    
}