Parsing text

[Layne Lund]

Okay, maybe this isn't an advanced question, but I'm not quite sure where to put it. In fact, my questions don't have to do with Java directly, but more on that later.

I'm working on a project that reads in a decent-sized RTF file and parses the content for information. I already figured out how to use the javax.swing.text package (and related subpackages) to obtain the content as a plain String. Now I'm trying to figure out how to parse this String to obtain the information that I want. Basically, the text contains multiple records that are all in a fairly uniform format:

<###> of <###> DOCUMENTS
 
Superior Information Services, L.L.C.  
 
* * * THIS DATA IS FOR INFORMATIONAL PURPOSES ONLY * * *  
 
JUDGMENTS FROM NEW JERSEY
 SUPERIOR COURT
 

 

Debtor: <first name> [<middle initial>] <last name> [; <first name> [<middle initial>] <last name> ...]
 
Debtor Address: <first name> <middle initial> <last name>
                <street address>
                <city>, <state> <zip>
                [SSN/EIN: <SSN or EIN numbers>]
 
                [<first name> <middle initial> <last name>
                <street address>
                <city>, <state> <zip>
                [SSN/EIN: <SSN or EIN numbers>]]
 
                [...]
 
Debtor Attorney: <attorney's firm (freeform text)>
 
Creditor: <name (free form text)>
 
[Creditor Address: <name>
                <address1>
[<address2>]
                <city>, <state> <zip>]
 
Creditor Attorney: <attorney's firm (freeform text)>
 
[Civil Suit Number: <civil suit number>]
 
Number: <number>
 
Vendor Number: <vendor number>
 
Date: <MM/dd/yyyy>
 
Filing Type: <filing type>
 
Case Type: <case type>
 
Amount: $ <#,###>
 
Status: <status>
 
County Filed: <county>
 
Place Filed: <place>
 
Description: <description>
 
[Remarks: <remarks>]

Let me describe my meta-language before I go any further:
< > encloses a description of the text
[ ] encloses optional text
... means that the pattern can repeat

My first thought is to write a lexer and parser similar to what I learned about in my compilers class. The above description would work fairly well as a grammar, I think. If I need to, I can change it into BNF even. My first question is would it be worth my time to download JavaCC or something similar to help write the lexer and parser? Ultimately, the program I am writing should be able to parse multiple documents with varying formats. Does JavaCC support multiple grammars?

Whether I use a tool like JavaCC or roll my own parser, there are a few complications:

1) In some situations, new lines have some significance. This is especially true for addresses. The only way I know how to tell where the City, State ZIP starts is by looking for a new line character. Usually a parser ignores whitespace, though, so I'm not sure how to deal with this. I would like to obtain the first two address lines, city, state, and zip separately, but I'm not entirely sure how. The optional second line in the address also complicates things. Does anyone have a suggestion here?

2) Should I view labels like "Debtor Address:" as a single token or as two tokens? I'm not sure which would be best/easiest to implement. It might not matter if I'm using a compiler-compiler, but I would like to know if anyone has suggestions here.

I will greatly appreciate any input anyone has.

Thanks,

Layne

[Stan James]

I learned text parsing (and programming) in a line-oriented world so I'd be tempted to read a line at a time, test for the easy tags like "Creditor:" or "Creditor Address:" and rely on position in a sequence for the others like the first few lines. That would probably be lots of code and fragile relative to changes or variations in format. I'll look forward to more sophisticated ideas!

[William Brogden]

I think I would read the whole thing into a String[] - then locate the groups of 1 or more lines that go with a particular marker such as "Debtor Address:" and send them to a method specific to each data type. In other words, separate the tasks of locating the data from interpreting the data.
The specific methods would not have to worry about detecting the end of the data type.
Bill

[Layne Lund]

Originally posted by William Brogden:
I think I would read the whole thing into a String[] - then locate the groups of 1 or more lines that go with a particular marker such as "Debtor Address:" and send them to a method specific to each data type. In other words, separate the tasks of locating the data from interpreting the data.
The specific methods would not have to worry about detecting the end of the data type.
Bill

That's actually very similar to what I've decided to try next. At the moment, I'm putting all the text into a single String rather than using a String[]. Then I'm trying to use regular experessions and the java.util.regex package to locate the data. I plan on writing individual methods to parse the individual blocks of data from there.

But now I'm running into trouble with the regular expressions stuff. I am starting by just trying to match the "<###> OF <###> DOCUMENTS" that occurs at the beginning of each record. Here's is the method that does most of the work:

    public List<CaseRecord> parse(String filename) throws IOException, IllegalFormatException {
        List<CaseRecord> caseList = new ArrayList<CaseRecord>();
        
        String content = this.readRTFFile(filename);
        Pattern p = Pattern.compile(this.getCaseRecordRegEx());
        Matcher m = p.matcher(content);
        
        System.out.println("HERE!");
        System.out.println("regex=" + p.pattern());
        
        while(m.find()) {
            System.out.println("IN LOOP!");
        
            int start = m.start();
            int end = m.end();
            
            System.out.println("************************");
            System.out.println(content.substring(start, end));
        }
        
        return caseList;
    }

The readRTFFile() uses the javax.swing.text.rtf to read the RTF file and return the contents as a String. I could easily split() this into a String[] if I want to. This method is actually in an abstract base class because there are at least two slightly different file formats. (This might end up being unneccessary because I may be able to deal with the slight differences if I write my regular experssion just the right way. But I'll figure that out later after I figure out what's wrong with my current regex.) At the moment, the subclass that I'm writing for testing returns the following regular experssion:

private static final String REGEX = "\\d+ OF \\d+ DOCUMENTS";

The two SOPs before the while loop print out (including the regex as I expect it). However, m.find() must be returning false because the SOP at the beginning of the loop doesn't print. This is where I'm stumped. I also printed the content String and it looks fine. So why am I not getting a match for my regex? Any ideas?

Thanks for your time to read my questions. I sure hope someone has some suggestions that can help me fix this latest problem.

Regards,

Layne

[Layne Lund]

Okay, I'm just retarded. After some experimenting, I found that the document contains the word "of" in the first line, but my regex was looking for "OF" instead. Once I fixed that, I was able to incrementally build a regex that matches the whole record!

Thanks for your comments Stan and William. Even though I didn't use them directly, they helped me start thinking about other ways to do it.

Regards,

Layne

[Stan James]

For grins, look at how Fitnesse parses Wiki markup. I'll see if I can say this so it makes sense:

There is a handler for every tag
Each handler has a public static final REGEX that matches the tag
The parser goes through all registered handlers
   Adds each handler to a set
   Concatenates the REGEX from each handler into a giant expression
Get the complete input text
Match the giant regex
while match is true
   iterate the match groups to find one that is not null
   the index of the match group matches the index of the handler
   get the handler from the set
   the handler replaces the markup tag with new text
   match again

I copied this scheme for my Wiki and it works pretty slick. Since each REGEX finds the smallest possible match it handles nested tags nicely from the inside out. You're not replacing (tho you could delete text) so this might not work, but it's worth an evening to read Fitnesse no matter what.