• Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Jeanne Boyarsky
  • Ron McLeod
  • Paul Clapham
  • Liutauras Vilda
Sheriffs:
  • paul wheaton
  • Rob Spoor
  • Devaka Cooray
Saloon Keepers:
  • Stephan van Hulst
  • Tim Holloway
  • Carey Brown
  • Frits Walraven
  • Tim Moores
Bartenders:
  • Mikalai Zaikin

Store attachments in Mongodb

 
Ranch Hand
Posts: 41
10
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Not many people will be interested in this. Or should be interested. The arguments on the web are plenty, and the general conclusion is that the best way to store files is on the system, in subfolders, with the database storing relevant filepaths and necessary meta-information. This is how JForum ihandles attachments.

Sane, but expressive, discussion,

http://www.onlineaspect.com/2007/07/10/image-storage-database-or-file-system/

Let's say, for my use-case,. I've heeded the warnings, yet database storage makes sense. How easy is this to do in JForum? Well, it must be done with source code, of course, not a binary. It is also difficult if the structure of JForum is not understood (or, like me, you are not aware of Java idioms).

That said, this post may be interesting for the insight into JForum's structure as much as any other reason.

Steps
Install and run a Mongodb database
Instructions on the Mongodb site are very good.

Get the Java driver, its a downloadable .jar, and add into your website library (WebcContent/WEB-INF/lib).

Load some preferences by a methods roughly parallel to that of JForum
Create this file,

WebContent/WEB-INF/config/database/mongodb_attachments/mongodb_attachments.properties

and populate, keeping the properties namespaced. No doubt this can be done smarter (but less clearly) by creating Mongodb properties then loading the config for two databases. But a separated config avoids difficulties in modding JForum's config file loading.

A Mongodb database definition is not as SQL. It has no need of pooling or concurrency, which is handled by the DB connection, which is multithreaded.
It can be run by default without authentication. It also has a special url string (I've broken this string to stop breakout on the forum, but it should be one continuous line).

}

You may be wondering, can Mongodb's lifesaving anonymous login default be used with blank username and password entries? Or will the code fail on the residual ":@"? Answer: the code will fail.

Add host, port, username, password, max pool size and timeout preferences. That's all that is needed.


During preload, JForum converts preferences into classes carrying those preferences. This code action is also used to select appropriate database settings (via properties in SystemGlobal.properties). The new Mongodb settings can be used as JForum uses them by altering,

/util/preferences/ConfigKeys.java

adding extra property mappings,


Unless the coder is dead-set on getting a close match to the SQL mechanism, this is all that is necessary.


Though attachment data and code could easily be hardcoded, as it is an append to existing code, I added extra overall config. Maybe it will help jog my memory sometime in the future. So to,

WebContent/WEB-INF/config/SystemGlobals.properties

I added,


Start the database
In,

/src/net/jforum/JForum.java

in startApplication(),


Now all the setup is in place to load basic preferences.

Build a new connection class
This is substantially different to SQL, and cannot use JForum's highly refined abstractions.

I settled for a whole new class, not part of JForum structure,

/src/net/jforum/MongodbConnection.java

This roughly mimics the current Connection.java file. As this is for Mongodb, it can be considerably simpler than the usual connection (integrating Mongodb into JForum's Connect.java file would require some strategic thinking (more than I was prepared to give to this).

Note there is no init(), a raw disconnect, and no connection instance return - that's Mongodb for you. 'databaseUp' is preserved in case some function like 'ping' is re-enabled. And it is a real class, not abstract, as it is for Mongodb only. Also note that it returns a Mongo.DB, not the MongoClient (this is a rough analogy to the behaviour of a SQL db connection).

I'm sure other can do better than this. But I am not in a position to really know if the new Enumerated Singleton's would be a better bet, or have data on other ways of coding, so here's mine,


Get the connection up and running
The DB connection work is done in the class JForum, not JForumBaseServlet. So, in,

net/JForum.java

in init() add,


This needs a new version of ForumStartupException. Code tests the connection return for truth (JForum's SQL connections catch while attaching to the ExecutionContext). Then add this to destroy(),


The references in the code should all be familiar from earlier.

After this, JForum loads a reference to the main DBConnection onto the ExecutionContext. I suppose this may be a preferred way to move a reference round, but if the DB is known to be Mongodb, there is no need.

Well, that's that, it's up and running. Now let's do something with it.


Modifying attachment handling
Uploaded files are handled by an Apache class. Since we want to stream the file data into a database, something other than what is provided is needed. For JForum's interface to the Apache code, goto,

/src/net/jforum/view/forum/common/UploadUtils.java

Don't delete saveUploadedFile() because it is also used for avatars, etc.
I added a method returning an InputStream from the uploaded FileItems.


Using the connection for attachments
There's plenty of abstractions in JForum to get lost in, but the only classes to study are

AttachmentCommon.java

which does physical movement of files into the web application, and the class tree,

  • attachmentDAO
  • GenericAttachmentDAO


  • Which write meta-information on attachments into the database. A reference to an instance of a DB-particularised instance of this is carried in AttachmentCommon instances.

    Either set of code can be modified, depending which is view taken (does AttachmentCommon 'handle' the attachments, or is the import a 'database write'?). I went for AttachmentCommon.

    Either way, the file copying must be removed from AttachmentCommon. I added some custom image crushing/thumb-creation code too.

    To use AttachmentCommon, replace the file transfer code. In insertAttachments(),


    'baos' is a reference to a ByteArrayOutputStream used to output from the FileItems. Then in editAttachments(), replace all delete code with,


    Note: the ids from the SQL database records are reused. Mongo does't care much if ids are continuous.

    That's all. Not much trouble, for covering all attachment CRUD.

    Tidy up
    You may wish to remove the now pointless column data from the SQL table 'jforum_attach_desc_seq'. These two columns are redundant,

  • physical_filename VARCHAR(255) NOT NULL,
  • real_filename VARCHAR(255) NOT NULL,


  • Though tidier, this cascades consequences. The generic SQL queries will need modifying, and some template code.


    Write a servlet for the upload
    Little title, long work. Sorry. It's a stock servlet, though. The web.xml item could look like this,

    The servlet might initialise, in init(),



    Then in doGet(), grab a URL, check for digits, then do something like,


    My images are guaranteed small. If the images may exceed, say, 256k, use GridFS code, not the above, and stream out (but in that scenario you must work much harder to justify database storage over JForum's built-in solution).
     
    Don't get me started about those stupid light bulbs.
    reply
      Bookmark Topic Watch Topic
    • New Topic