Native XML database

[Valentin Crettaz]

I have a big XML file (>100MB) that contains a great deal of knowledge. My application uses that knowledge and makes it available to the user. The fact that the data is contained in an XML file is completely transparent to the user. What I would like to have is some kind of XML native database (as a component, for instance) that I could embed in my Java application. I would like to store the big XML file in that database so that queries could be performed very quickly. I have tried various RDBMS ans OODBMS solutions but the results were not very satisfying.
I really want to go with a native XML database for storing my XML file because I have good reasons to believe that such a database provides the fastest access to elements contained in an XML file and the lowest memory footprint for the application. Simply doing XPath queries on the XML file will not solve my memory concern.
I've had some interactions with Software AG whose Tamino XML Server is a great native XML database, but there is no way to embed it in a standalone application.
I have tried Ozone, eXist and Xindice but they are too slow. I'm also trying Berkeley DB XML but when a query is supposed to return a list of elements, it simply returns one XmlValue of type NODELIST that has to be re-parsed in memory, and thus, defeats the whole point of having a database for lowering the memory footprint.
I've gone through this list of native XML databases but I can't find the right solution.
Does anyone know a good native XML database solution that could solve my problem? What I need is simply an embeddable native XML database.
[ December 10, 2003: Message edited by: Valentin Crettaz ]

[Gregg Bolinger]

If the XML native DBMS's are not working for you, why are you still looking for that kind of solution? Why not use something like HSQLDB for an embedded DB in your application? Just curious why your solution HAS to be XML based?

[William Brogden]

How complex are the queries that must be made in this database?
If they are simple and predictable, why not code them in Java directly.
Bill

[Valentin Crettaz]

Originally posted by Gregg Bolinger:
If the XML native DBMS's are not working for you, why are you still looking for that kind of solution? Why not use something like HSQLDB for an embedded DB in your application? Just curious why your solution HAS to be XML based?

I have tried several RDBMS and OODBMS solutions. However, these solutions were not meant to store XML natively, they were just *trying* to store XML natively in a OO or relational way. This caused lots of unneeded overhead and this is the reason why I'm looking into *real* XML native database solutions.
My application has to be XML-based because I'm storing all the knowledge in an XML file.

Originally posted by William Brogden:
How complex are the queries that must be made in this database?
If they are simple and predictable, why not code them in Java directly.
Bill

Queries are not that complex, I could hardcode them in Java directly. However, I have different *unrelated* knowledge bases encoded in XML and I would like to use the same querying API above it whatever the contained knowledge is.

[William Brogden]

I've gone through this list of native XML databases but I can't find the right solution.

Wow, thats a huge list. Are you trying to stick to an industry "standard" like XQuery or XPath?
Software for querying large text databases is still a hobby of mine -in fact it was at one time my business - so I am really interested in following up on this problem. Could you perhaps describe a typical search and what the result should look like?
Bill

[Axel Janssen]

here is an overview about native xml databases:
http://www.rpbourret.com/xml/XMLDatabaseProds.htm#native
In my company they are using xindixe and eXist. But I am not very involved in this stuff.
EXist seems to be a little bit more stable. First they've used xindice, but there was lots of complaints about lack of documentation and general bugginess.
Exist looks better.
Query language is XPath.
XQuery would be better, but there seems to be no open source implementation (from what I've heard, might be wrong).
Axel

[Valentin Crettaz]

Bill, it's good to know that other people have the same problems/interests in this new area
I have tried the following open-source solutions up to now:
ozone: OO persistent DOM (Java)

Problem 1: XML storage in an OODBMS requires an additional conversion layer => slower than native solution

Problem 2: Does not support the storage of large documents

Advantage: Supports XPath

XDB: RDBMS (Java and C++)

Problem: XML storage in an RDBMS requires an additional conversion layer => slower than native solution

Advantage: Supports XPath, XQuery

Xindice: Proprietary storage (Java)

Problem: Does not support the storage of large documents

Advantage: Supports XPath, XUpdate

Berkeley: Native XML DB (Java and C++)

Problem 1: XPath support is limited

Problem 2: No support for XUpdate and XQuery

Advantage 1: Java API = JINI layer to a powerful C++ API

Advantage 2: Is embeddable

Advantage 3: Supports XPath

Tamino: Proprietary (native) XML storage (Java and C++)

Problem: Is not embeddable

Advantage: Supports XPath (extended), XQuery

I'm currently looking into eXist, a native persistent DOM (Java)
But the problems are the following:

Problem 1: XML storage in an OODBMS requires an additional conversion layer => slower than native solution

Problem 2: Does not support the storage of large documents

As Gregg proposed, I'm also looking into HSQLDB (Thanks Gregg )
Axel, thank you for the list but this is the same as the one I provided in my first post
[ December 17, 2003: Message edited by: Valentin Crettaz ]