Granny's Programming Pearls
"inside of every large program is a small program struggling to get out"
JavaRanch.com/granny.jsp
Win a copy of GANs in ActionE this week in the AI forum
or WebAssembly in Action in the JavaScript forum!

Thomas Whitmore

Ranch Hand
+ Follow
since Aug 05, 2004
Cows and Likes
Cows
Total received
0
In last 30 days
0
Total given
0
Likes
Total received
0
Received in last 30 days
0
Total given
0
Given in last 30 days
0
Forums and Threads
Scavenger Hunt
expand Ranch Hand Scavenger Hunt
expand Greenhorn Scavenger Hunt

Recent posts by Thomas Whitmore

Hi Siyaram,

Our PowerMap JDO product has specific features for database integration and transparency to existing RDBMS/ applications. These are standard capabilities and can be used within Java, Tomcat and servlet, or J2EE containers.

Enterprise Edition also includes a JCA resource adapter. This is an optional developer tool, enabling automatic transactioning & enlistment within the J2EE transaction.

Most projects just use the 'explicit' approach and request transactions from code, this seems to be the more popular technique as as most shops do not require the RA.

Let me know if this helps,

Regards,
Thomas Whitmore
www.powermapjdo.com
Hi,

I had some problems with FireFox still caching content (pages not images) after I had set the cache header.

I think the answer was, that it was already in the cache and just refreshing wouldn't update the headers which said the cached data was still good. Try Ctrl-R for Reload and/ or clearing FireFox's cache and it should then pick up the no-cache headers after that.

Let me know if this helps.


Cheers,
Thomas
www.powermapjdo.com
14 years ago
Hi Damian, Jeroen,

Depends on your database. MySQL and other databases will definitely discard connections after a period of inactivity. This is a bugger for code which would otherwise be correct to cache database connections.

Jeroen makes a point -- keeping a DB Connection per servlet, is not usual practice. This probably wouldn't work correctly for threaded requests, once servlet was under load. Separate connections would typically be used for separate requests.

Single connection might be shared for 'singular' services such as key allocators etc, the problem here is that there is no portable api or SQL to check a connection is still good. The most performant technique is to detect failed queries and update the connection, but this is more complex to code. Most session activity (as opposed to keying/ management) still uses separate connections.

Our JDO product uses this technique, and the solution performs well. Let us know if this helps,


Cheers,
Thomas Whitmore
www.powermapjdo.com
14 years ago
Hi Cheng, people,

Let's say I've a method that takes in a String representation of a date & then parses it to return a Timestamp object. In the midst of parsing, exceptions could be thrown (due to null string, invalid format, etc).


This is a tricky question, as is most error-handling.

1) For survivability and tolerance of minor user mistakes, you want the code to continue executing with a reasonable default (null).

2) For formal testing, correctness, error diagnosis & resolution you want to report the problem.

Most Java approaches seem to follow one side or the other, but don't really fit both these requirements. eg the assert keyword is good for formal testing but bad for survivability in production.

The solution I came up with was this:
1) log a WARNING thru log4j
2) continue with a default value
3) monitor logging at warn or error levels, and assert no unexpected warning or errors at the end of each unit test.

Is this what you were thinking?


Cheers,
Thomas Whitmore
www.powermapjdo.com
[ March 08, 2005: Message edited by: Thomas Whitmore ]
14 years ago
How about we use Data Access Services as the term rather than DAO. DAO refers originally to one-object-per-managed-instance 'peers' which are rather unnecessary.

What you really want is a Service Facade like eg.

CustomerMgr
Customer getCustomer (Object id)
Object getCustomerId (Customer cust)
void storeCustomer (Customer cust)
void deleteCustomer (Customer cust)
...
List searchByName (String prefix)
List listByRegion (String region)
...

Service Facade can be implemented with JDO, JDBC, HB or literally *any* backend behind it. For JDO or similar managed solution the methods are pretty much one-liners, but this is all it takes to achieve the full modularity and design benefits.

The principle here, is KISS - all you need is a point of control where you can control caching or application requirements, change implementation etc.

And this is the exact correct design to implement these, when and should you need to. Until then, it's just a direct logical handle on what the app requirements.


Cheers,
Thomas
www.powermapjdo.com
Hello Durga, this is a good question because it is fundamental.

> My doubt is regarding the controlling and the general flow.
> Its necessary to have just one servlet controller that takes the request
> parameters from one jsp and delegate it to some other jsp.

Controller concept is to handle the user's Interactions with the page. Form submission, query/ edit and navigation, as well as simply requesting pages.

Different areas of data (News, Searches, Edit forms, Static Content) would have different Controllers requirements. Implementation could be under any name or place but you what a class is for.

> But when there are thousands of jsp's and there is only one controller

I would refer you to the Spring framework which uses meaningful names for meaningful concepts. 'Handler' or 'Interceptor' may be used as early stage of request handling to handle generic Login, Menus, templating etc before delegating to the Controller which handles the user's major task.

If there are thousands of jsps then they are 1000x similar templated content and similar behaviour. Or there are multiple Controllers.

> then how does the servlet controller knows from which jsp the request is
> comming and which page to render next.

Let's take a sidestep: Meaningful designs might have StaticPageController, [customer data] SearchController, [customer data] EditController, ForumPageController, IntroPageController, ...

Here's a tricky one, for knowing where you are on the Web: what about the URL path?

As for coming or going, that's also a good question. The answer is fundamental: it depends on whether you need to do anything or not. If your apoplication is passive, let HTML do the navigation for you; you controller only has to return the right static content.

If your application has to take some action, then the controller has to handle this and flow or redirect onwards. You guessed it, form submissions and interaction.

So if you structure your data areas to be requested into paths, eg /cust/search, /cust/edit, /order/search etc, and implement Controller functionality sensibly for these, you'll probably have a good idea of where you are and what you're doing...

as well as a fairly sensible software design.

As to exactly which customer, search criteria, selection etc you are working with -- nobody's found a better use for URL parameters. For each page when you generate action and navigation hotlinks in the JSP you output the data to parameterize the request.

Lastly, paths are useful for document areas. The machine uses paths in the file system so /docs/* might go to DocumentController which sources content from eg. the file system.


Anyway, check out the Spring framework. I recommend Tomcat/ Spring/ JDO as an effective combination. You want to understand clean and direct application code to get the job done. Hope this helps,


Regards,
Thomas Whitmore
www.powermapjdo.com
14 years ago
JDO
Hi Punna,

JDO is good. You want to check out the new JDO 2 releases, as these add features like Fetch control and advanced Querying which were previously non-standard.

Sun's JDO-RI is not a real usable JDO implementation, it does not support RDBMS or multi-user requirements. It's more an example and resource for JDO vendors to work from.

JDO 2 is fully comparable with Hibernate, and may have some advantages -- better handling of == operator, standardization, clean separation of model from storage, etc.

Hope this helps.

Cheers,
Thomas
www.powermapjdo.com
Hi Chris, PJ, TIm, people,

Interesting discussion. I'm not going to pretend to solve any performance problems, but this application should be eminently do-able.

PJ's solution is nice because you can see it. And touch it, in the form of monolithic blocks of code. And modify it manually. All 750 classes worth of it...

But I suspect that when you start doing reporting, or want to tune which fields the app fetches, that those monolithic blocks don't have flexibility in this area. Or if they do, it's starting to get as complicated as a real mapping layer anyway.

What does it end up with, DAO count = databases * fetch plan count * table count = 3 * 250 * 4... ? The tangible nature of the solution probably turns out to be its own limitation; this is why we wanted to get away from expressing this stuff as code in the first place.

[Admittedly PJ expresses his stuff with an intermediate tool. If the intermediate tool is so good, why do we even need to see the Java though ?]

Mapping a field value or a primary key is trivial; it's the structure and control to achieve good efficiency that are the challenge.

Anyway, a few thoughts on performance.

Chris's app has ~100,000 rows in some tables. Answer: if they're indexed, table size is essentially free well into the millions of rows. Most index pages will live in cache since a whole index should only take a few Mb.

App has over 250 database tables. Question: are a lot of these Unioned data, or split off by date or period? Number of tables is also free as far as performance goes, but mapping may be a little outside the norm.

We would have under 300 concurrent users. Answer: many standard databases & hardware should be able to execute 30 - 120 structured reads per second, and perhaps 15 - 30 writes. By structured, I mean an Order, 5 OrderLines and their Product references.

I'm not sure how fast your 300 users type exactly :-) but I suspect they take more than 10 seconds per transaction. Allow the application a duty ceiling of 30 - 40%, so it still seem responsive.

App is going to be deployed on several different brand of database. Answer: here's your customer requirement right here, mapping is probably going to be the ongoing issue rather than performance.

Most actual SQL is a stored procedure it is going to be hell. Answer: yes. And if the customer's DBA requires SPROCs you're going to have to work with these anyway, sorry.

Scaleability and caching. Answer: yes. Most O/R tools have a shared instance cache which provides basic performance boosts. Much further gains are typically available, if required, from application-aware caching.

Basically Caching is where you should deploy your Java code, not a Data Access Service but an Application Cache Service.

This is written to udnerstand the application's data locality and be able to cache groups/ sets which the application commonly requests. A naive instance cache can't help much here, but your ProductCache can remember Products by Category, TransactionCache allows Transactions to be held for hot (TimePeriod, CostCentre) sectors, or whatever your app works by.

Even large and high-volume tables, far too large to cache in their entirety, have 'hot spots' which are 20% of the size but 80% of the activity. A smart cache can be much more effective with these. Try and code the cache to understand locality and grouping, rather than dictating the actual access pattern. This way you can adapt well to reporting or different access patterns.

How large do you want to scale anyway ?


Cheers,
Thomas
www.powermapjdo.com
Hi people,

After much delay, the JCP committee has voted to approve JDO 2 !

The work is still in progress but JDO 2 has already released a final draft and brings the standard up to full-weight application requirements - reporting, detached insts, fetch plans etc...

We're also celebrating the JDO 2 specification. If you want to learn about JDO for free, or compare new JDO features, visit our website for more details. (It's a limited time thing).

As widely discussed, the incoming JDO2 features are continuing to push the benchmark in O/R tools for the Java developer. Some of the more exciting features include :-

Attach/ Detach - detachment enables use or modifying data between requests, for example in an HttpSession. This is particualrly useful for web apps.

Fetch Plans - big increases in performance and database efficiency, as application can now control what fields must be retrieved for a given usecase. Fetch Groups are defined in metadata making tuning easy.

Query Projections - projections enable JDO Queries to retrieve related data, columns for reporting, or fetch subordinate tree-structures in a single pass. This handles navigation, mapping and SQL generation and will run smart optimizations on your JDBC code, too.

Aggregates & Grouping - these extend JDO's query features to the full extent of SQL's capabilities. This improvement brings JDO to being comprehensively & efficiently able to execute any standard report, without the hard work writing SQL (and tuning it, and re-writing it...).


This is a pretty big step forward, folks. Of course there are other fish in the sea, but this brings the JDO standard up to some serious application tasks with ongoing support and addition of some EJB features over the next two years.

Looking great! It's time for a tuna-and-mayonnaise sandwich :-)


Cheers,
Thomas
www.powermapjdo.com
Hi Ilja, people,


Let's get that straight: It offers, for example,
- a keyboard shortcut to show a tree of all the classes where a polymorphic method is implemented, including the option to open those classes;
- the ability to switch the blocks of an if-else statement with just a couple of key presses, without changing its semantic;
- instant, automatic compilation every time you save a file, even if it contains compile time errors;
- instant, in-editor diff between the file in the editor and the version in the repository,


I know, Eclipse is just *so* powerful as well as being extremely fast. Having worked with the internals, I am enormously impressed with IBM's engineering.

Avoidance of Swing is a real performance boost. Compared with JBuilder and NetBeans, much of the useful assistance features (eg, code completion) appear much faster, as well as far more elegant.

No need for manual mounting of packages, it just works and 2-3x faster. Technologies such as cached workspace info and close parser/ compiler integration are probably at work here.

Quick tip: I like to designate the 'workspace' to be a short pathname close to the root - eg D:\workspace or D:\Dev\workspace. This helps interoperation of multiple tools/ IDEs which live below D:\Dev, with multiple projects which live in Workspace.

True, the basic Eclipse download is a 'vanilla' platform with only IDE and Java coding features. Yet this vanilla platform outperforms most other offerings. When you want JSP, XML etc editing you can add these - either as plugins or with a package such as WebSphereAD built on the eclipse platform.

As far UI designers, I've found it far more effective to use a static helper class to generate GridBagConstraints, and keep my code clean, than it ever was to edit or maintain the vast ugly wadding produced by a Swing designer.

YMMV, but I'm getting 40 miles to the gallon with 600 horsepower :-)


Cheers,
Thomas Whitmore
www.powermapjdo.com
JDO
Hi Priya,

JDO is the most modern & popular standard for Java data access & persistent storage. Basically, it handles the chores of database access and mapping so you can concentrate on requirements and using your object model.

This is the most natural and productive way to develop with Java or OO languages. Previous approaches such as JDBC and BMP have had excessive overheads due to the focus on coding data access rather than actual application. JDO also improves over historical CMP approaches in many ways.

Great question about the container! JDO provides management of persistent data on a per-session basis (PersistenceManager). Independent changes eg for separate web clients can be handled separately. JDO handles this simply and correctly, making it clearly visible what should be going on, whereas CMP relies a little too much on 'magic technology' making it much harder to debug what should be going on inside the appserver.

JDO can be run inside or outside the container and can also be used with lightweight web servers eg Tomcat. These lightweight options are often preferred as being much easier for developers to test, debug and deploy business functionality.

Test/ debug cycles can for example be 3 - 5x more productive, with business logic able to be debugged outside the container using simple JUnit tests. Even logic destined for use in a J2EE container, can be debugged separately if implemented with JDO, making the QC cycle a lot easier for developers.

The future JSR-220 standard is a proposed step to merge the benefits of JDO technology onto EJB. Customers have been requesting better EJB facilities for years now and this would be great but ratification of this step would still be a year or more down the road after JDO2.

Hope this helps.


Cheers,
Thomas
www.powermapjdo.com
Hi Andles,

Interesting question. We use JDO but the same fundamental logic applies.

Businesses and business systems require responsibility for actions and data, and normally designate this in a heirarchical structure.

This means that most of the first-class entities with which a business deals primarily, will be structured with 1-many rather than many-many relationships.

Exceptions to this are relatively infrequent in business modelling. Route planning and multiple transport or connection paths is the one which springs to mind. This would be used where eg transport connections needed to be modelled, but there was no 1st-class need of management, responsibility or other data; just a reference to the opposite end.

(Actually most distribution or logistics applications are immediately going to want costs or congestion-delay data on their connections. This would require this to be promoted to a 1st-class entity anyway, obviating the many-many relationship).

Hope this helps.


Regards,
Thomas
www.powermapjdo.com
Hi Benjamin,

Great project. What you are describing sounds like either A) a full-text database application, or B) a full-text search of filesystem data. Either of these would work depending on one or multiple concurrent clients. Either of these would perform depending on building & storing indexes for the search.

Filtering text CDATA from XML is easy to code, but you're correct that this would be inefficient to process repeatedly. Either DBMS or filesystem approach should manage this process so as to perform it only once upon document loading/ modification.

The Database option does have advantages in allowing concurrent access & maintenance of the document library. Separate columns (CLOB or BLOB) would store the XML version and perhaps also the 'stripped' plain text, but the main requirement would be storage of Lucene's index data. The issue here would be concurrent access and whether index data could be stored/ updated by document chunks, or as one monolithic central index (more difficult for concurrent updates).

I'm not sure what the advantages of an XML database would be as text-search is the particular challenge and XML databases wouldn't offer any extra help in this regard.

Hope this helps.


Regards,
Thomas Whitmore
www.powermapjdo.com
Hi David, Lilly,

We've had great results with the Spring Framework also using in our case JDO for database access.

For Web applications, wiring-up of business services and controller objects, and portable configuration, Spring is the greatest thing since sliced bread. :-)

Spring also offers a couple of approaches to wrap data-access code, as David mentions. Our decision was to retain a vanilla approach and request a session, only when we needed a session. Spring greatly simplified the wiring of the application and almost halved our time-to-release.


Cheers,
Thomas
www.powermapjdo.com
Hi Vinnie,

Congratulations and you're probably having a fair bit of fun coming over. Spring is great fun if you're building web apps; the first and best point of it is, a convenient and maintainable way to hook up and configure all the different parts of your app.

Business services, http request controllers, db access, mail senders, product catalogs, app data... all the config can be done bean-style and easily maintained... all the inter-object references can be set by Spring...

Thus eliminating the need for setup constructors, passing refs from here to there, push- and pull- updating, order-of-init problems, a whole couple of categories of code whose only logical function was to be a place bugs could exist, gone in one fell swoop.

First off, the vast array of choices is staggering and I'm completely unsure where to go from here. I have been using JDBC and "mapping" my own objects manually but we all know how un-maintainable that is.

Well, we all know that it's a fair bit more maintainable than scattering JDBC code everywhere. But a proper O/R engine will give you field mapping performance *and* enable you to run different access patterns/ queries etc across these, to fetch your app data.

Anyhow, here are my questions and assumptions, please answer/correct me where necessary? I'm not looking for a flame-war and a bunch of sales pitches based on everyone's favorite technologies. What I really need is a seriously objective look at what I'm asking, please!

JDO is cool, Hibernate is cool, which one you might use depends on what level of project you're doing and whether you want to go standard or not.

Personally I like proper support for JOINs and UNIONs -- and I've been suprised to find the number of different impls which will map a union but execute it is as repeated non-unioned queries. ?

Proper support for unions, and certain column/ data types, needs control down to individual column specs. Most data of course is happy with default, but I like tools which can actually deal properly with the database structures that are out there.

How efficient is Hibernate vs. using, say, stored procedures and a DAO/JDBC framework? I know many programmers answer questions like this emotionally and entirely based on preference, damn the consequences. Again, objectivity please. I'm actually quite skilled with RDBMS (sql server, firebird, others...) and am a fan of stored procedures, used apprpriately and if there is a benefit to using them.

One of the advantages of JDO is that it uses compiled data access and keeps 'pure' object identities. HB uses reflection and proxies quite a lot and these impose some amount of overhead. Probably ok for smaller apps though.

Do Hibernate and JDO overlap entirely or are there differences? Which is better? Ahem, by better I mean, which is the most proven and highly-used, highly-acclaimed of the two? Which is the most *simple* to work with?

JDO is a Java standard and gives you some degree of pluggability between vendors. New JDO2 query technology can also bring some advanced performance capabilities. HB is obviously free and has a loyal fan base.

6. Is there a way to auto-generate the descriptor files for my objects based on my database schema w/ either JDO or Hibernate? Can this be done within NetBeans? (Open Source or Freeware only!)

Commercial impls have Reverse Engineering and UI tools with various degrees of sophistication. Freeware typically doesn't get to that standard.

As for IDEs, I mainly do Java and web coding, but damn if Eclipse ain't it. Perceptually about 3x faster and a whole lot slicker. IBM engineers nailed this one, hard.

You'll also want to check out Ant. At some stage you'll want to start building/ deploying libraries, apps or other production artifacts. After you've jarred or copied filesets manually a few times, you can setup Ant to do this for you. Not just the convenience, but it will do it reliably.

Ant tip: clean by moving files to a 'deleted' bin, rather than deleting directly. Your buildfile will likely have errors until you've tested it, and I've had deletes go off in my project root folders... :-( :-( :-(

Doesn't having to update descriptor xml files and business objects *sort of* cancel out many of the benefits of O/R mapping?

Well, except I can template up a getter/ setter class, add it as JDO, Forward Engineer it to a table, clean up a few fields/ mappings and add a row to the key allocator table -- in about 8 minutes.

Versus 2 or more hours.

Will EJB 3.0 blow the competition out of the water? I have read only small bits about it but it looks promising. I haven't bothered to learn much about EJB in its current state as I am advised not to around every turn!

Well, forgive my cynicism but it would *have* to look promising at this point. But the success stories I'm hearing these days are multi-headed Tomcat with JDO or HB for storage.

EJB3 of course is being hyped and that's not just J2EE but a 'new' persistence spec, the big box vendors were probably starting to feel left behind, this persistence spec will also be usable outside J2EE so that's what most of the hype is about.

Does Spring overlap with JDO and Hibernate? Spring seems like a jack-of-all-trades and I'm unsure how it would be useful to me, exactly.

You won't even have to code constructors since Java defaults a no-arg one if no others are specd. Nor init code. All your init becomes XML and you can see and edit it in the one place.

I know I may be asking some fairly simple questions but believe me, I've been researching the topic for a few weeks now and am ready to being using one of these technologies in future projects, some small business open source apps I'd like to release next year, if possible.

Interesting. Web interface or client/ server? Half of Spring's usefulness is the Web MVC stuff. Solving the config- and init-code problem is a pretty good bonus for any app, though. I think Spring can be be used in many parts, and would entirely be usable for this - but to be honest web is what I'm using it for.

If anyone can show me some SIMPLE code examples of Hibernate performing some of the common tasks you would normally use SQL for, I'd REALLY appreciate it. I found their documentation to be rather complicated, at first glance, and haven't had much luck finding straight-up, simple code to demonstrate.

:-)

Certain people have been prone to spitting the dummy at other apis, despite nothing being perfect and hb having some distinct technical tripups.

But what you might be seeing that could confuse you, are chained invocations -- these are quite a useful feature where methods that could have been declared to return void, actually return the object you called them on. This allows you to conveniently call several modifiers or methods in a chain.


Cheers,
Thomas Whitmore
www.powermapjdo.com