Which parser to choose for MIME multipart/mixed

[Tien Shan]

I want to use a MIME parser to parse a file like the following, which is not an email, nor it is http request body.

MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="MIME_boundary-1"; type="text/xml";
Content-Description: Header info

--MIME_boundary-1
Content-Type: text/xml; charset="UTF-8"
Content-Transfer-Encoding: 8bit
Content-Location: Header.xml
<?xml version="1.0" encoding="UTF-8"?>
<Header>
  <dataPoints1>400</dataPoints1>
  <dataPoints2>500</dataPoints2>
   .. other items..
<Header>

--MIME_boundary-1
Content-Type: multipart/related; boundary="MIME_boundary-2";type="text/xml";
Content-Description: subHeader with binary data

--MIME_boundary-2
Content-Type: text/xml; charset="UTF-8"
Content-Location: subHeader1.xml
<subHeader1>
  <processingTime>1234567890</processingTime>
  .. other items..
<subHeader1>

--MIME_boundary-2
Content-Type: application/octet-stream
Content-Location: data1.bin
10101000011111100000000000 (example)

--MIME_boundary-2 
Content-Type: application/octet-stream
Content-Location: data2.bin
01000100011000011110000001111 (example)

--MIME_boundary-2 
.. subHeader2 start ....

The file has one header section, which is in XML format and several sections whose first part is subHeader and second part is binary data.
That is, section immediately after "Content-Location: dataX.bin" is binary data section. The length of this binary data is given inside the XML header ("dataPoints").
Each subHeader section has information specific to the binary data that follow.

Once I parse the XML header part, I know how to extract values from that section.
I want to parse and store the binary data (let's say "data1.bin") into an array for further processing. I am not sure how I will after I get to this point but I want to get to that point first.
(The binary data is encoded in IEEE754 float- 32 bit single precision, I need to convert those bunches to "normal looking numbers"- even though they have 10 to 12 digits and do all sorts of calculation.)

Many samples I read in the internet are for emails.
Google has given me javaMail, mime4j, mimeEntity of IBM/Lotus domino, netscape.messaging.mime.
I am thinking to use mime4j. Or should I use javamail?

There are different types of binary data sections for each subHeader, all properly separated by "--MIME_boundary-2"s, from which I will only need, say "data1.bin", so a blanket reading of all content is not efficient. That's why I want to use a parser and write proper codes (in Java) that basically grabs what is required to grab.

I will have to read 50-60 files like that, so a faster implementation is ideal.

I found a sample at mozgoweb.com/posts/how-to-parse-mime-message-using-mime4j-library/ but it parses emails only.
I will keep on googling, but with the kind of skill I have, I doubt I will find a right one and understand.
Can someone give some idea, an outline, some code snippets if possible, how I read and get those parts.

Thank you for your time reading and possibly giving some hints.

Cheers!

[Tien Shan]

In Stackoverflow, the.jxc has written:

Get the "Boundary" and split the message on lines matching "--". Then for each chunks, parse headings until you get to CRLF, CRLF. Then your content starts.

Easy for gurus. Greek for me.
[I do see "0A" "0A" at the end of the "dataX.bin", before beginning the binary part, when I use a Hex editor (FelxHex or HexNeo). That is probably "LF".]

[Tien Shan]

I found a site that was like manna for me.
I was able to extract header info, and the mime parts.
I don't know if I am allowed to link this website, so I will just give the name of the file- "SimpleMimeReader.java".

A HUGE part of my worries are gone. JR, Thank you so much.

I will need to modify a section of his code.
My next struggle is how to extract a particular "part" when the Content-Type is multipart/related.
Right now, as it is, the SimpleMimeReader.java extracts all the constituent parts, say, in one block, because it does not
distinguish boundaries inside boundaries. (The secondary boundary being, "MIME_boundary-2")

[Tony Docherty]

Thanks for posting your solution so far.
Yes you can post relevant links to other sites but the class name should be enough for any interested party to find the site.

If you get stuck on the next section post your code and the details and we will try to help.

[Tien Shan]

Tony:
Thanks for the words of encouragements. Yes, I have made a lot of progress since the last post.
To extract information from within the big "part", which had a boundary like "MIME_boundary-2", all I had to do was to call the SimpleMimeReader.java again, but setting boundary as "--MIME_boundary-2".
Next, I added some lines here and there, basically skipping "parts" that I didn't need. Works beautifully. I am grateful to the author of that program.

My next challenge is how to read/parse a file (or rather, a stream, which if written to a file, would be an xml file) that has not been written.

Here is the scene.

At the very beginning, the above mentioned parser gets me a header section. The header is written in XML. In fact, for test purposes, I wrote that extracted header XML into a file, successfully read it using a DOM parser and extracted what I needed. I can write the files now because I am in test phase. I *do* need information from the headers, but I don't want to write header files for every file I process. There will be many dozens of files, it may slow down processing. (I can probably delete the files after I read it, but if I can get away by not physically writing the files, that would be something!)

That is what I meant by reading a file without writing it.

Here are some snippets how I created header (XML) file, using the SimpleMimeReader.

SimpleMimeReader smr = new SimpleMimeReader(new FileInputStream(my_original_headerXML_plus_binaryFile));
ByteArrayOutputStream baos = new ByteArrayOutputStream();
while (smr.nextPart()) { //this method makes it possible to get data from a MIME file
    getPartData(baos);
    //fol. 2 lines pseudocode
    if (header part contains "multipart/related"), it's body
    else it is header
       baos.writeTo(new FileOutputStream(fileName)); //this is how/where header file (XML) is created
}

Somewhere else, I have an xml reader code, snippets:

DocumentBuilderFactory dbFactory = DocumentBuilderFactory.newInstance();
DocumentBuilder dBuilder = dbFactory.newDocumentBuilder();
Document doc = dBuilder.parse(xmlFile); //this xmlFile is the one I just created above, of course, after new File-ing the fileName string.
 
//optional, but recommended
doc.getDocumentElement().normalize();
	 
//get attribute of root element
Element projectPath = doc.getDocumentElement();
String pr = projectName.getAttribute("projectName");
//.. do something with pr.. and so on 

What would you suggest?

[Tien Shan]

Looks like I am not going to get answers.
In the meantime, I have no choice but to create xml header files and later delete using file.delete() command from within the program.
Time to close this thread and move ahead.
Thanks everyone and Cheers!

[Tien Shan]

Where is "Close this thread" button? Can't see.

[Tony Docherty]

Tien Shan wrote:Looks like I am not going to get answers.
In the meantime, I have no choice but to create xml header files and later delete using file.delete() command from within the program.
Time to close this thread and move ahead.
Thanks everyone and Cheers!

Sorry for the lack of response, I had marked the thread as resolved after you had posted the solution and hadn't spotted you had posted again.

Reading directly from a stream rather than a file should be simple. The SimpleMimeReader class has a constructor which takes an InputStream as an argument, ie the one you are currently using when you create a FileInputStream object to read the file. So if you have an InputStream for reading the header file just substitute the FileInputStream for it.

[Tien Shan]

Hi TD,
It worked alright, just as you wrote. I was scared even without trying!

One reason I was hesitant was that I am using ByteArrayOutputStream (baos) to write the header information to a file (an xml file).
All I had to do was convert that baos to ByteArrayInputStream- and pass that input stream.

Some approximate code:

Before:
Document doc = DocumentBuilder.parse(xmlFile);

After:
ByteArrayInputStream bais = new ByteArrayInputStream(baos.toByteArray());
Document doc = DocumentBuilder.parse(bais);

Cannot ask for more!

Now, please close this thread or tell me how to do it.

Cheers.

[Tony Docherty]

Glad to hear your problem is now resolved.

As I said earlier the thread is already marked as resolved which is probably why you can't see an icon to close the thread.