• Post Reply Bookmark Topic Watch Topic
  • New Topic

java.lang.OutOfMemoryError: Java heap space using Stringbuilder  RSS feed

 
Tanishka chandrasekaran
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi ,

Iam getting a file from a URL and i want to store the contents of the file to a string in order to pass to the JSON Object.
so iam trying to do these steps where iam getting an Out of memory exception.

InputStream is = new URL(url).openStream();
BufferedReader rd = new BufferedReader(new InputStreamReader(is,Charset.forName("UTF-8")));
String jsonText = readAll(rd);
JSONObject json = new JSONObject(jsonText);

In the readAll method i have written the code as below:

private static String readAll(Reader rd) throws IOException {
StringBuilder sb = new StringBuilder();
int cp;
while ((cp = rd.read()) != -1) {
sb.append((char) cp);
}
return sb.toString();
}


Iam getting an Out of memory exception at the line sb.append((char) cp);
i need to get the string in order to pass to Json Object and similarly avoid this Memory issue. The file from the url getting passed is really huge file. So thats the problem.Please help in handling the same and any alternate methods for this approach.

 
Surendra Kumar
Ranch Hand
Posts: 236
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
OutOfMemoryException is bound to happen if you're dealing with a huge file.

Does reading a line at a time and converting to JSON work for you?

If not, try increasing the JVM heap memory (-Xms, -Xmx).
 
Tanishka chandrasekaran
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What do you suggest here .. I need to basically pass the file contents to the JSON object to make JSon read and parse further. Instead of Stringbuilder ,do we have any alternate approach.
 
Surendra Kumar
Ranch Hand
Posts: 236
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Can you read the file one line at a time and convert it to JSON object, instead of reading the whole file at a time?
(This way, you're not creating big string)
 
Tanishka chandrasekaran
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
can you please give me sample code for same. How to change the existing code of mine?
 
fred rosenberger
lowercase baba
Bartender
Posts: 12562
49
Chrome Java Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Tanishka chandrasekaran wrote:can you please give me sample code for same. How to change the existing code of mine?

First, we don't do that here. We don't provide solutions. We will help you do it, but you have to show some effort yourself.

Next, I would suggest you StopCoding. You have not answered Surendra Kumar's question. Is it possible to process the file a line at a time? That is not something we can answer, since we don't know anything about your data.

Until you answer that, there is no reason to try changing anything.
 
Tanishka chandrasekaran
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Sorry for not replying to the earlier question. Yes i can read the file one line at a time and convert it to JSON object . But i want to put the complete JsonObject to JsonArray for further processing.
 
Paul Clapham
Sheriff
Posts: 22819
43
Eclipse IDE Firefox Browser MySQL Database
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I would guess that if you don't have enough memory to put the file contents into memory, then you aren't going to have enough memory to wrap those contents with JSON and put that into memory either.

Of course "pass the file contents to the JSON object" is rather incompletely specified; I'm assuming that you want something like



but there are many other things you might want. And if what you want involves not using everything in the file, there might be a solution.
 
Rob Spoor
Sheriff
Posts: 21131
87
Chrome Eclipse IDE Java Windows
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
So use a different constructor:
This will probably read the file only in parts, and construct the JSONObject on the fly.
 
Tanishka chandrasekaran
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
what is wrong in this line of code:
InputStream is = new URL(url).openStream();
JsonReader rdr = Json.createReader(is);

the code gets stuck with above line of jsonreader.
 
Tanishka chandrasekaran
Greenhorn
Posts: 14
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
i have changed my code something like this:

My json file have to be read from a url through http . once i read the url ,open the url stream and i have to read line by line . its a json format file . Please help.
i have tried to read from the url as below:

InputStream is = new URL(url).openStream();
BufferedReader reader = new BufferedReader(new InputStreamReader(
is, Charset.forName("UTF-8")));
String line, results = "";
while ((line = reader.readLine()) != null) {
results += line;
}
reader.close();
is.close();
JSONObject json = new JSONObject(results);
JSONArray fileArray = json.getJSONArray("Documents");

then iam looping the Array again for each line to perform some actions.Any suggestions to improve the code here.
 
Alvin Parker
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Have you looked at how much memory your program is consuming? If you have a lot of system memory available and the JSON object is not larger than the memory you have left over, why not just add your compiler arguments (something like this would increase max memory to 8 GB if you have it 'java -Xmx8G -cp PATH_TO_CLASSES com.yourpackage.YourJSONObject ')? If you are running via Eclipse or some IDE, you can go into your project properties and add the -Xmx compiler argument so you have more memory for testing. If you can't do any of that, you must have a truly massive JSON file in which case you might have to look for a strategy where you can serialize the bytes and make smaller JSON objects perhaps with an object that maps them together. But your easiest solution seems to be just to try the compiler args.

Also, the "code getting stuck" is not enough info to tell you what is wrong with that line. If it's a really big file, you would expect it to take a while to process the URL InputStream, right? If the program is freezing up with no errors being shown (and no out of memory), I've solved issues in the past like that by adding timeouts to my read functions. Essentially, I put it in a Thread and have the Thread sleep for 10 MS once it process some buffer. For instance, save 1 MB of byte[] sleep for 10 MS, save 1 MB sleep for 10 MS... until done.
 
It is sorta covered in the JavaRanch Style Guide.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!