• Post Reply Bookmark Topic Watch Topic
  • New Topic

Writing images via ImageIO eventually fails with no error  RSS feed

 
Alvin Parker
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I have a project that makes video frames from a number of overlayed images and then writes those frames to disk (e.g. img_001.png, img_002.png...). Once done, I execute FFmpeg as a CLI to convert the images and wav file to a MP4. This works fine for video that is a couple of minutes long. But, at 30 frames a second, you can imagine the size of my image directory.

Given that this can be very memory intensive, I have created two Threads to manage the frame creation - one makes the frame as a BufferedImage and stores the final frame to a List<BufferedImage>. The other thread gets BufferedImages from the List and saves each frame out to disk intermittently. It sets the BufferedImage in the List to null once saved so that the List does not change size while the creation Thread is working and so that memory can be freed when the image is no longer needed in memory. No matter how large my videos are, the memory footprint appears to never get above 350 MB so the strategy seems to work fine where memory is concerned.

Both Threads have timeout cycles. I allow them to work for 20 MS and then sleep them for differing intervals (it takes longer to save a frame than make one).

For videos of 5,000 frames or less, this process seems to work fine. But, above that number it fails. The program does not crash but my logs reveal that the FrameSaverThread is no longer saving frames while the FrameMakerThread chugs along still making frames and adding them to the List.

Any thoughts on what I can do to ensure that ImageIO and my Thread do their work to the end would be much appreciated. Since I'm not getting any error messages and it works almost every time, I'm having troubles understanding where to begin to solve this problem. Here's the code for where the frames are saved :
 
Ulf Dittmer
Rancher
Posts: 42972
73
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Given that this can be very memory intensive, I have created two Threads to manage the frame creation

I don't follow that rationale - using threads does nothing for memory consumption, but it introduces the overhead and synchronization issues of concurrent execution. Have you tried saving each image as soon as it's created? Or does that cause other issues?
 
Alvin Parker
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Ulf.

I suppose I should clarify. I have two Threads for more than just dealing with memory, but that was a major consideration - although I could certainly be wrong. The maker Thread worries with nothing other than just making the frames and adding to the buffer. If I saved once all the frames are made, I would use a lot of memory and would be limited to how large my videos could be. The saver Thread keeps this concern at bay by saving at its own pace and clearing out each image's buffer space as it saves but does not block FrameMakerThread so FrameMakerThread can go at whatever pace it's able to manage without concern for where we are in saving video frames at any given time.

Maybe what I need to do is follow more of your suggestion and make FrameMakerThread responsible for kicking of a save method each time and then using recursion. I could design it so that it's all async - it might be faster that way too. I am just avoiding re-writing any part of the program if others have experienced something similar and might have an easy suggestion. I'm making a streaming server for them right now and once done this entire process changes and so I don't want to do more to it than I absolutely have to right now.
 
Steve Luke
Bartender
Posts: 4181
22
IntelliJ IDE Java Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
My guess is that this is a synchronization issue: specifically in this line: for(; frameNum < frames.size(); frameNum++). If frames is the List holding the images to write then it is possible that the writing thread doesn't see the correct count of frames in the list. Even if the reader thread wrote 100 frames, the writer may only see 10 published to its thread. It then processes those ten, finds it is now at the end of the list, and stops. Sleeping and what not do not help, you need synchronization barriers.

The List is the wrong collection to use here. What you should use is a Queue implementation, probably a BlockingQueue. The reader would be putting frames at the end of the queue and the writer would be popping them off the front of the queue. The two most notable BlockingQueues are LinkedBlockingQueue and ArrayBlockingQueue, the difference between them being if it is bound in size or not (if you are worried about memory footprint you might choose to use an ArrayBlockingQueue with a size limit large enough to fill your memory. Then if your reader far outpaces your writer the reader will eventually slow down and pause to let the writer catch up. Not sure if this is desirable. The alternative is the queue can grow very large and eventually run out of memory.)

Another alternative that is really very much the same, is have the reader create a writing Task (a Runnable which will save the image) which it submits to an ThreadPoolExecutorService to run. You could then set a reasonable number of threads writing at the same time. You have to be careful with this though, because if you have multiple threads writing to the same disk it often is slower than a single thread doing it. So unless you have a RAID you might consider using a single writer instead.
 
Steve Luke
Bartender
Posts: 4181
22
IntelliJ IDE Java Python
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just wanted to add: What I posted above was a guess, hopefully a guideline to get you pointed towards a solution. But programming based on guesses is a good way to paint yourself into a corner. What you should do is
1) Add logging statements, see what code gets executed for each loop, and what the values for the list size, the counter, etc... Also log when the method gets in the exception handler and when it is completing.
2) Attached a debugger and see the thread states when the writing stops. Is the writing thread blocked? What on? Is it dead? This can help lead you to the exact cause, not just a guess.
 
Alvin Parker
Ranch Hand
Posts: 39
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Those are fantastic thoughts. I'm sorry it took me so long to come back in here. I am on another project that is consuming me. I am glad to come back here and find such a great answer.

To accomplish this, I'm going to change my Threads to Runnables and add them to the Executors framework. Before starting this project, I had no Threads. Then I had only one Thread. Now I have two Threads and it seems the Executor framework will manage them better. To be sure, I can simply make a for loop, have executor execute one, inspect a Future and then have it execute the next one which will keep everything really async. Or, I could use the join() method, but either way I'll be guaranteed an async flow (right) with assurances that if a Thread has a problem I won't be left in the lurch.

What you describe seems right. There are still hundreds of images to process and yet the Thread thinks it's at the end of the List so goes ahead and terminates.
 
Jim Venolia
Ranch Hand
Posts: 312
2
Chrome Linux VI Editor
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Putting on my embedded software hat, I'd declare "BufferedImage imageToSave" before the for loop, then assign to it over and over. That should just reuse the object, as opposed to constantly creating new ones, which gives the garbage collector a lot of work.

If you think resource usage may be the problem then instead of logging I would declare a variable, and set it to assorted values as the code runs. Set to 1 before ImageIO.write, 2 after the call, 3 after the if(written), 4 in the else, 5 before getGraphics.dispose(), etc. Note in C that variable needs to be declared volatile, lest the compiler decide "Hey, this variable isn't used, I think I'll ignore it". Comment out the LOGGER calls. Hmmm, I'd prolly also declare 2 variables, goodWrite and badWrite, incrementing each as appropriate in the loop.

Uhhh, why are you setting imageToSave to null?

But I digress. Build some way into your system such that when it misbehaves you can dump the value of the that variable. Under *nix I setup a signal handler, when the program dies send it that signal. No clue how to do anything like that under Java.

I think having 2 threads is the correct way to do it, along with giving the queue connecting them a relatively small size (get it working with 10 or so, then when the code is working tune the size), and having the producer block when the queue is fall and the consumer block when it's mt.

The elephant in the room is, is the video coming in in real time, or are you reading it from something?

 
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!