Yes, -Xmx sets the maximum size of the JVM (The default value is 64MB). The "reasonable" size is determined by the requirements of your program (don't set to a gig if you only need 256MB) as well as the physical RAM on your machine. Java Command Documentation
You could just keep increasing -Xmx until the program works. If there's no major changes to the data sets or code, you should be fine. Java has built in tools to monitor resources so you should be able to get an idea as to how much is being used and if the -Xmx setting is way out of line.