I'll address the issues separately to avoid confusion.
1. Is
Java heap memory same as memory shown in an OS task manager?
No. That's the memory allocated by OS to JVM, not just for Java objects created by an application, but also for storing class metadata and data structures used by the JVM itself.
Heap memory is the sub portion of this memory set aside by the JVM for java object creation.
Whether JVM releases the memory when its heap reduces depends on the JVM's implementation.
For example, Sun/Oracle JVM has command line args like Xmx, Xms and HeapFreeRatio. Whether JVM releases OS allocated memory when its heap reduces depends on these settings.
2. Is there a leak in JavaFX, specifically, in Group.getChildren().clear()?
I tried your program and I see used heap reducing just fine in every case after removing points.
In some cases, I triggered a GC manually, in others it was triggered by JVM itself. See attached image.
In my opinion, there is no leak. If there was one, that used heap line is not going to drop like that.
But if you're convinced otherwise, file a bug against JavaFX with a heap dump.
3. JMonkeyEngine behaviour
From JMonkeyEngine source code, I get the impression that it does quite a bit of active memory management itself, using techniques like weak references and even triggering System.gc().
I don't know if JavaFX too does it internally, but aleast it's one possible reason why JMonkeyEngine seems more memory efficient.
Also, check what the Xmx and other such settings are for the other app which uses JMonkeyEngine.
4.
...in JavaFX this does not happen and with every group of points added the system runs slower.What to do ?
- Try setting Xms and Xmx to the same value and to a high value, so that time is not wasted on memory allocation and garbage collection.
- To see if slowness is due to garbage collection, set the command line flag to dump gc activity (I'm not sure what this is on Mac. If this is Oracle JVM or OpenJDK JVM, it's -verbose:gc.
- Too many nodes in scene graph is a well known javafx bottleneck. See if you can move away from scene graph and instead use Canvas, but this means your application will have to handle quite a bit of the rendering logic.
- There is always the possibility that javafx is simply the wrong tool for this app.