MediaMetadataRetriever for frame retrieval from mp4

[Jim Harrison]

Hello,

I'm developing this for the Droid X2 (Android 2.3.3) and came across some issues with trying to retrieve the bitmaps from the video.

The following code produces the same frame from the video even though the video is 7 seconds long. I only get one saved IMG file.

Obviously there wouldn't be 7000 unique frames but I would have expected there to be 20-30 IMG files.

I would greatly appreciate any direction as I believe I've followed the API.

        MediaMetadataRetriever mediaMetadataRetriever = new MediaMetadataRetriever();

        mediaMetadataRetriever.setDataSource(_currentVideoFile.getAbsolutePath());
        String METADATA_KEY_DURATION = mediaMetadataRetriever
                .extractMetadata(MediaMetadataRetriever.METADATA_KEY_DURATION);

        Bitmap bmpOriginal = mediaMetadataRetriever.getFrameAtTime(0);
        int bmpVideoHeight = bmpOriginal.getHeight();
        int bmpVideoWidth = bmpOriginal.getWidth();

        Log.d(LOGTAG, "bmpVideoWidth:'" + bmpVideoWidth + "'  bmpVideoHeight:'" + bmpVideoHeight + "'");

        byte [] lastSavedByteArray = new byte[0];

        float factor = 0.20f;
        int scaleWidth = (int) ( (float) bmpVideoWidth * factor );
        int scaleHeight = (int) ( (float) bmpVideoHeight * factor );
        int max = (int) Long.parseLong(METADATA_KEY_DURATION);
        for ( int index = 0 ; index < max ; index++ )
        {

            bmpOriginal = mediaMetadataRetriever.getFrameAtTime(index * 1000000, MediaMetadataRetriever.OPTION_CLOSEST);

            bmpVideoHeight = bmpOriginal == null ? -1 : bmpOriginal.getHeight();

            bmpVideoWidth = bmpOriginal == null ? -1 : bmpOriginal.getWidth();

            int byteCount = bmpOriginal.getWidth() * bmpOriginal.getHeight() * 4;
            ByteBuffer tmpByteBuffer = ByteBuffer.allocate(byteCount);
            bmpOriginal.copyPixelsToBuffer(tmpByteBuffer);
            byte [] tmpByteArray = tmpByteBuffer.array();

            if ( !Arrays.equals(tmpByteArray, lastSavedByteArray))
            {
                    int quality = 100;

                    File outputFile = new File(mediaStorageDir , "IMG_" + ( index + 1 )
                            + "_" + max + "_quality_" + quality + "_w" + scaleWidth + "_h" + scaleHeight + ".png");

                    OutputStream outputStream = new FileOutputStream(outputFile);

                    Bitmap bmpScaledSize = Bitmap.createScaledBitmap(bmpOriginal, scaleWidth, scaleHeight, false);

                    bmpScaledSize.compress(CompressFormat.PNG, quality, outputStream);

                    outputStream.close();

                    lastSavedByteArray = tmpByteArray;
            }
        }

        mediaMetadataRetriever.release();

Thanks
Jim

[D. Smith]

Just checking references here, but what is the actual value of the String variable METADATA_KEY_DURATION when you extract it and max after you parse it into an int? It's not very likely, but since the variable name and the key constant have the same name I want to make sure you aren't getting scoping problems. Also, can you verify that the loop itself is running more than once?

[Jim Harrison]

The value for METADATA_KEY_DURATION (as the String) is 7100 (presumably milliseconds). There is no issue w/ scoping as the parameter passed into the method is scoped to MediaMetadataRetriever while the variable getting assigned is scoped to the method.

Yes the for loop is executing that many times: I have a Log.d statement that I see.

Thanks
Jim

[D. Smith]

A couple thoughts:

Since presumably the value of max is 7100 (no errors parsing into an int), then only the first seven iterations of the loop would actually pull a frame time inside the video's duration. With index = 8, the requested frame time is already past the video duration at 8 seconds (8,000,000us), index = 10 -> 10 seconds, etc. So all subsequent 7090 iterations probably continuously pull the last readable frame; perhaps try and change the requested time so you get better granularity of presentation time values (i.e. so all values of the loop are within the duration). Technically, though, we would expect at least a few of the first seven requests to be different frames (unless the video really has very few sync frames in it), so...

Another thing that has given me issues in the past is trying to use the OPTION_CLOSEST flag; it seems that some device implementations are unable to read anything but sync frames out of encoded media types. Perhaps try a different flag that specifically only reads back the sync frames from the content.

So in other words, try modifying the frame request like so:

//Index is in ms, multiply by 1000 to get us.  Read closest sync frame to requested time
bmpOriginal = mediaMetadataRetriever.getFrameAtTime(index * 1000, MediaMetadataRetriever.OPTION_CLOSEST_SYNC);

[Jim Harrison]

Hello,

For my original code, yes, I would have expected there to be more than just one image saved.

Thanks for the advice. After the code change, unfortunately, same as before, it only saved the first frame.

My loop now looks like the following:

        ....
        final int stepping = 1000 / 24;
        ....
        for ( int index = 0 ; index < max ; index += stepping )
        {
            ....
            bmpOriginal = mediaMetadataRetriever.getFrameAtTime(index * 1000,  MediaMetadataRetriever.OPTION_CLOSEST_SYNC);
            ....
        }
        ....

By the way I have also tried the following with the same results of just one image:

            bmpOriginal = mediaMetadataRetriever.getFrameAtTime(index * 1000, MediaMetadataRetriever.OPTION_PREVIOUS_SYNC);

            bmpOriginal = mediaMetadataRetriever.getFrameAtTime(index * 1000);

Any other options to try?

Thanks
Jim

[D. Smith]

I have taken and run your original code, along with the modifications you made, on my Nexus S device and everything basically works as expected. Depending on the options selected, a 14s test video produces around 24 images (OPTION_CLOSEST_SYNC) or 240 images (OPTION_CLOSEST). I do not have Droid X2 hardware, but I also ran this code in the Motorola Droid X emulator, which uses the same software image so the performance should be comparable, and the results where the same there as well.

This leads me to believe that the issue lies either with the video you are reading or with that device, because I cannot find any further fault with your code and it seems to be working from the tests I've run.

[Jim Harrison]

Thanks.

I just shot 2 more videos: 4-5 seconds each. One is done w/ the phone in portrait and one w/ phone in landscape. For me, both still produced the same results.

They are either my left or right hand counting to 3 or 4. JavaRanch doesn't appear to support a single file that is 13megs (.7z)

Can you post your video that I can try? Or can I post my video for you to download and try?

Thanks
Jim

[D. Smith]

The video sample I just pulled from the web, the file is at the following link: http://dev.exiv2.org/attachments/345/04072012033.mp4

That same site has other video files as well: http://dev.exiv2.org/boards/3/topics/1189

[Jim Harrison]

I just downloaded your referenced .mp4 file.

My code does not create any IMG files.

Is the concept of "sync frame" a setting on phones/cameras? I just looked on my Droid X2 (v2.3.3) for settings and anywhere else I can think of and can't find a reference to setting it or not.

I searched google for "droid x2 sync frame" and couldn't see anything glared about this feature not working/etc.

Thanks
Jim

[D. Smith]

It's not a programmable thing, sync frames are a concept of encoding video. Encoded video files don't store every video frame of data to keep the file size down. They only save specific frames known as "key frames" or "sync frames" as a full frame, and then just the pixel deltas for frames in between. This is why, by default, sync frames are all the retriever returns; it's easier to find them and extra decoding doesn't need to be done. How many sync frames are inserted in the video file is up to the application that encoded it (on the desktop, it's usually a programmable setting of the encoding software) and I'm sure it's a fixed value on the device.

It's theoretically possible that a video file have only the initial frame fully rendered, although unlikely as playback would be very slow. However, having issues with the MP4 I used rules out the camera's recording capability as the problem.

My code does not create any IMG files.

Interesting, in your initial post you mentioned you were getting only one initial frame, now you are getting zero. I wonder how just a different movie file could affect things in that way...are you able to get your code working in the emulator?

[Jim Harrison]

My apologies!

I meant to say my code does not write more than 1 .png w/ your mp4 file that I downloaded.

Thanks
Jim