The easiest way to increase the JVM size of the app server in 5.1 is via the admin console.
Expand the Servers tree and click on Application Servers. In the app servers list, click on your app server name. Under additional properties, click on Process Definition. On the next screen, click on
Java Virtual Machine. On the next screen, you can change the initial heap size and maximum heap size. Make sure to press Save.
Fyi, this is for WAS 6.1, but provides some useful memory analysis discussion:
http://www.ibm.com/developerworks/websphere/library/techarticles/0706_sun/0706_sun.html