How to extract finance related terms/words from a given text using Java

[rastogi payam]

Hi All,

I have a PDF file which I parsed into text using PDFBox API, now I want to extract finance related terms/words and their frequency from that text file. I also googled for the same and found that we can use GATE/OpenNLP but didn't find any concrete example. Please help.

[Bear Bibeault]

What criteria are you using to determine if a term is "financial" or not?

[William Brogden]

As an initial exercise, try extracting all words and their frequency. Consider how to handle UPPER/lower case and plurals.

Natural Language Processing can get pretty tricky, best to start simple.

Bill

[rastogi payam]

Hi William,
Here is the code which I have written to extract and count the words from the text. What should be my next step. Please advice.

package com.example;

import java.io.File;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.Iterator;
import java.util.Map;
import java.util.Set;
import java.util.TreeMap;

import org.apache.pdfbox.pdmodel.PDDocument;
import org.apache.pdfbox.util.PDFTextStripper;

public class ReadPDFTest 
{
	public static void main(String args[])
	{
		try
		{
			File pathToFile = new File("some_pdf_file.pdf");
			PDDocument document = PDDocument.load(pathToFile);
			PDFTextStripper s = new PDFTextStripper();
			String content = s.getText(document);

			content = content.replaceAll("\\.", " ");
			content = content.replaceAll("\n", " ");
			content = content.replaceAll(":", " ");
			content = content.replaceAll(";", " ");
			content = content.replaceAll(",", " ");
			content = content.replaceAll("\""," ");
			content = content.replaceAll("\""," ");
			content = content.replaceAll("“", " ");
			content = content.replaceAll("”", " ");

			ArrayList<String> list = new ArrayList<String>();
			String[] strArr = content.split("\\s+");
			Map<String, Integer> map = new TreeMap<String, Integer>();
			for(int i=0 ;i<strArr.length;i++)
			{
				list.add(strArr[i]);
			}
			Collections.sort(list,ALPHABETICAL_ORDER);
			for(int i=0; i<list.size();i++)
			{
				String str = list.get(i);
				if( !map.containsKey(str) )
				{
					map.put(str, 1 );
				} 
				else 
				{ 
			        map.put(list.get(i), map.get(str) + 1 );
			    }
			}
			Set<String> set = map.keySet();
			Iterator<String> it = set.iterator();
			while(it.hasNext())
			{
				String str = it.next();
				System.out.println(str+" = "+map.get(str));
			}
		}
		catch(Exception e)
		{
			e.printStackTrace();
		}
	}
	
	private static Comparator<String> ALPHABETICAL_ORDER = new Comparator<String>() {
	    public int compare(String str1, String str2) {
	        int res = String.CASE_INSENSITIVE_ORDER.compare(str1, str2);
	        if (res == 0) {
	            res = str1.compareTo(str2);
	        }
	        return res;
	    }
	};
}

[fred rosenberger]

As Bear said, you need to determine what makes a word "financial". That has nothing to do with Java or writing code - it has to do with your brain and/or your specs.

[William Brogden]

For performance sake, your Map

  Map<String, Integer> map = new TreeMap<String, Integer>();  
         

should use an int array size 1 instead of Integer as the value. The reason being that Integer objects have no simple increment method - what you have written would require a new object creation every time. Incrementing the int[] at index 0 does not.

Assuming you have that parsing and counting code working, look at the output - do you see "financial" terms?

Bill

[Paul Clapham]

The java.util.concurrent.atomic.AtomicInteger class has methods to increment (etc.) the integer it contains.

[rastogi payam]

Yes I can see financial terms in the output along with general English terms. Now how can I differentiate between these two categories.

[Joanne Neal]

rastogi payam wrote:Yes I can see financial terms in the output along with general English terms. Now how can I differentiate between these two categories.

As Bear and Fred have said, that's for you to decide. If this is anything more than just an exercise then the design specification should tell you the answer. If it is just an exercise, then presumably you can just select your own terms.

Edit: Having just reread your post maybe that isn't what you meant. You say you can see the financial terms so presumably you know what they are and you just want to know how to separate them.
In that case you just need to put a check before the bit of code that adds the terms to the Map in your code. If it's a finacial term you add it, otherwise you don't. You could put all the financial terms in a Collection of some sort and then just check if the collection contains the term.

Collection financialterms = new ....
  .
  .
  .
if (Financialterms.contains(term)) {
    // Add it to map
}

[rajdeo singh]

First you need a dictionary that contains the financial term and then you need to classification of document using NLP.
may this link helpful to you.

http://www.eur.nl/edsc/english/databases/financial_databases/ and then you need to apply java code for this.