Total memory won't reduce after freeing memory allocated for String?

[Haiti Meid]

Hi,

I'm confused about this: I load a string from a big text file. After using it, I set the variable to null, then call System.gc(). The used memory reduces, but why not with Runtime.totalMemory()? For example:

# java -jar dist/string.jar 
                           max: 592.00 MB
                         total: 30.13 MB
                          free: 29.65 MB
                          used: 487.95 KB
after loading big file... (text = [...])
                           max: 592.00 MB
                         total: 481.69 MB
                          free: 70.56 MB
                          used: 411.13 MB
after setting variable to null...
                           max: 592.00 MB
                         total: 482.00 MB
                          free: 480.88 MB
                          used: 1.12 MB

I just post the result of test, since the code is simple:
- load text from file to variable (the size of text file is ~66.3 MB);
- print memory info;
- set variable to null;
- call System.gc();
- print memory info.

In the past I worked with Delphi. Loading big string from text file, the used memory increases; and after emptying the variable, the used memory returns normal.

Now in Java, is this related to JVM or OS, or my knowledge is bad?

Thank you so much. I'm waiting to hear from you.

Sincerely,
Haiti

[Jesper de Jong]

Welcome to the Ranch.

Java manages memory automatically, using a garbage collector. You don't have to free objects manually, and it decides by itself when it will clean up unreferenced objects. There is no way to force garbage collection; in particular, calling System.gc(); will not force garbage collection, it's only suggesting to the system that it might be a good idea to do garbage collection, but it does not mean that it will be done right away.

You should normally not try to manually tell the garbage collector to do its work; just let it do it automatically, because usually it knows best when and how to clean up the garbage. It can take seconds, or even minutes, before the garbage collector cleans up the unused objects, and that's not something to worry about.

[Haiti Meid]

Thank you... However, in my example, how can I free those ~452 MB of total memory? Because I need to run some instances of my application. But my Fedora just have about 1GB of RAM is free for applications to use. When I run 3 instances of my jar, Fedora hungs...

I'm sorry because currently I have a little of time to optimize my code and stay here regularly in hours to discuss more with you... I'd loved to do that since Java is my passion and I'm a beginner, I wish to learn a lot. At this moment, my question is just in my example: how can I free ~452 MB which was be increased by JVM? I post my example code here:

package string;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.FileReader;
import java.io.IOException;
import java.io.File;

public class Main {

  public static void main(String[] args) {
    Main main = new Main();
    main.printMemInfo();

    String text = main.testLoadBigFile(args.length == 0 ? null : new File(args[0]));
    System.out.println("after loading big file... (text = " + (text == null ? "null" : "[...]") + ")");
    main.printMemInfo();

    text = null;
    System.out.println("after setting variable to null...");
    System.gc();
    main.printMemInfo();
  }

  public String fileSizeToStr(double fileSize) {
    final short blockSize = 1024;
    final String displayUnits[] = {"bytes", "KB", "MB", "GB", "TB"};

    byte i = 0;
    while (true) {
      if ((fileSize < blockSize) || (i == (displayUnits.length - 1))) {
        if (i == 0) {
          return String.format("%.0f %s", fileSize, displayUnits[i]);
        } else {
          return String.format("%02.02f %s", fileSize, displayUnits[i]);
        }
      } else {
        fileSize /= blockSize;
      }
      i++;
    }
  }//fileSizeToStr()

  public String align(Object str) {
    return String.format("%30s", str);
  }

  private final Runtime runtime = Runtime.getRuntime();

  public void printMemInfo() {
    System.out.println(align("max") + ": " + fileSizeToStr(runtime.maxMemory()));
    System.out.println(align("total") + ": " + fileSizeToStr(runtime.totalMemory()));
    System.out.println(align("free") + ": " + fileSizeToStr(runtime.freeMemory()));
    System.out.println(align("used") + ": " + fileSizeToStr(runtime.totalMemory() - runtime.freeMemory()));
  }

  public String loadText(File file) throws FileNotFoundException, IOException {
    FileReader fr = new FileReader(file);
    BufferedReader br = new BufferedReader(fr);
    StringBuffer sb = new StringBuffer();
    try {
      try {
        while (true) {
          String line = br.readLine();
          if (line != null) {
            sb.append(line).append("\n");
          } else
            break;
        }
      } finally {
        br.close();
        fr.close();
      }

      return sb.toString();
    } finally {
      sb.setLength(0);
      sb = null;
    }
  }

  public String testLoadBigFile(File f) {
    try {
      return loadText(f);
    } catch (Exception ex) {
      return null;
    }
  }

}

I mean, when my application does not need lots of memory anymore, why does not JVM return it back to OS? How can I tell JVM to do that?

Just my idea as a reference, please do not mind because this is Java forum: In Windows and Delphi, everything is simple as I expect: the exe borrows memory from OS when it needs; and when done, it returns memory to OS, so other applications can use that memory. Even it's still running...

[Henry Wong]

Haiti Htd wrote:
I mean, when my application does not need lots of memory anymore, why does not JVM return it back to OS? How can I tell JVM to do that?

Basically, no C/C++ program that uses malloc() and free() will ever return memory back to the OS. The malloc() and free() function uses sbrk() which extends and shrinks the memory footprint as needed, but free() doesn't ever shrink it. In other words, freeing memory is simply marking the memory as freed so that it can be malloc'ed again later.

Having said that, this may no longer be true; maybe there are malloc and free implementations that can return memory (you should do research), but even then, there is no way to return memory free in the middle of the heap to the OS.


Anyway, back to the topic -- having worked with the JVM itself, I can say that the JVM uses malloc and free (and some stuff that sits on top of that), so no, it won't return the memory back to the OS.

Henry

[Christopher Nortje]

Hi you can't force the JVM to return the memory to the OS directly, you can change the min and max heap size that is allocated to that java instance.
Just look at the JVM arguments for your JVM example:
-Xms128m -Xmx256m (min and max heap size).
-Xms<memory_size> Sets the initial amount of memory allocated to the Java heap.
-Xmx<memory_size> Sets the maximum amount of memory allocated to the Java heap.
Please note that in your case you are using up memory and freeing the memory that has been allocated to the JVM not the OS directly.
It seems like you are starting up three different java processes, they all have memory allocated to them, they don't share that memory.
Can't you use the same java process and spawn new thread for each file that you would like to process?
Have a look at “Tuning Garbage Collection” for your relevant JVM.

[Haiti Meid]

Thank you all...

It's my bad knowledge about heap memory. (About Delphi, I remembered not clearly... Perhaps it was related to pagefile... I'm sorry because of telling without re-checking that).

I managed to let my application load file by parts. Now for loading every 5 MB of text (utf-8), the total memory reaches ~60 MB, it's good for my machine. And I can run even ten of instances :-)

Thank you again. You all helped me a lot.

@Christopher: Thanks, that's good idea. But the application is a console, I just have one System.out to print reports :-)

Sincerely,
Haiti

[Jesper de Jong]

Haiti Htd wrote:Thank you... However, in my example, how can I free those ~452 MB of total memory? Because I need to run some instances of my application. But my Fedora just have about 1GB of RAM is free for applications to use. When I run 3 instances of my jar, Fedora hungs...

Objects become eligible for garbage collection when no live threads in your program have a reference to them anymore. So, to make sure that the garbage collector will eventually free the memory, just drop the references to those objects. There's no way to tell the JVM "free these 452 MB" directly, but you don't need to. The garbage collector will clean up the objects automatically when some other process needs memory.

[Nico Van Brandt]

I wonder why it's possible to call System.gc();

To me it gives a wrong sense of safety

1) It's not said it will be executed within the next few seconds
2) It may not be the best time to perform a cleanup

[Jesper de Jong]

System.gc(); indeed doesn't do what many users assume it will do. It's only useful in very special circumstances when you know what your particular JVM implementation will do when you call it. You should only call this when, after research etc., you know it does anything useful for your particular case.

Never call this only because you have some vague feeling that it might help (that would be superstitious programming!). It might even work against you and make performance worse in some circumstances.

[Christopher Nortje]

But the application is a console, I just have one System.out to print reports :-)

Why don't you just write to the OutputStream directly instead of building up a StringBuffer. If you’re not manipulating the String you’re just wasting memory.

    public void writeText(File file,OutputStream out) throws FileNotFoundException, IOException {
        FileReader fr = new FileReader(file);
        BufferedReader br = new BufferedReader(fr);
        try {
            for(int n = br.read(); n != -1 ; n = br.read()){
                out.write(n);
            }
        } finally {
            br.close();
            fr.close();
        }
    }

Something like this should drastically reduce your memory requirements. You could pass System.out as the OutputStream parameter.
Before I forget make sure that you synchronize the write to System.out, you don’t what anther thread writing to System.out while you’re in the middle of writing a document.