Android Question Camera Preview Memory Leak?

[BertI]

I've been playing around with the camera preview function via CamerEx e.g to transmit camera video via a LAN (like CCTV client example). However I find that my app crashes after some hours. It seemed that the app RAM usage just kept increasing (as monitored via native Process Stats function), reaching around 200Mb on this particular tablet (running Android 4.4.2) before it crashed.

As a check I compiled and ran Erel's CCTV client app:
https://www.b4x.com/android/forum/threads/android-based-closed-circuit-tv-cctv-example.23601/

This exhibited similar behaviour. I.e. After a few hours crashed and I could see RAM usage gradually creeping up in process stats until the ~200Mb crash point.

Trying the same code on a somewhat more powerful tablet (running Android 4.4.4) exhibited much less of a problem. This has been running for days and although I can still see a small memory creep it is nowhere near as much.

I am definitely not an expert so purely conjecture - could it be that on the slower tablet the preview event is internallly firing more often than it can be serviced by the call back and somehow the lost frames are accumulating somewhere? Or any other suggestions?

 

[Erel]

Going over the code, I don't see any clear memory leak.

If you plan to run this for many days then you will need to restart the app every few hours or it will eventually fail. Too many things happen in Android for it to run infinitely.

The way I solved it with the UI Cloud devices (which run for months) is by creating a small app that is always running in the background and is responsible for restarting the main app.
The main app should close the activity and kill itself with ExitApplication after a few hours.
The background app sends an intent that starts the main app every 30 seconds. If it is already running then nothing will happen, otherwise it will be started.

 

[BertI]

Thanks for the quick response

From the difference in symptoms between the two tablets my gut feeling was also that there is nothing wrong in the top level code. So my suspicion was that it might be somewhere in the part of Android that delivers the frames upwards. Sorry for such a loose description but like I say I'm on a slow and sporadic learning curve.

I did try at one point to put an incrementing counter in the Camera_preview event in the CameraEX class and decremented this counter after the jpeg conversion of the frame in the main activity Camera1_preview sub. I also set the Interval to 10ms to maximise the possible rate of jpeg conversions. In my mind this was to test the idea that the CameraEX Preview event was being fired more regularly than the main activity could respond to (i.e wondered whether a large queue was thus building up - if such a mechanism exists). However the counter remained at zero. So I then wondered if it could be something lower down i.e where Android might buffer the frames before these are then finally 'consumed' by the main activity. Again don't know if such structures/mechanisms exist, so vague conjecture.

I'm pretty sure its not in the communications side of things either because I also took these out of the equation but still the same symptom.

I guess you are probably right about Android in general, though I have to say that before I started introducing the camera code I had a fairly complicated (and Im sure badly written) app using various UI and communication structures which has been running for quite a while (few months) with no apparent memory creep and no need so far for a restart. Maybe fortunate pick of code components and perhaps environment, but encouraged by this.

In the case of using a background app could that also mean that this itself would have to be occasionally restarted (say by the main app)? Have to admit not so keen on the idea of invoking watchdogs on a regular basis since each one could be a glitch in the user experience. So still interested in understanding the root cause. Guess some more dabbling and learning required

 

[Erel]

In the case of using a background app could that also mean that this itself would have to be occasionally restarted (say by the main app)?

You can use StartServiceAt to schedule this app to start every few minutes. Again, nothing will happen if the intent is fired while the app is already running.

 

[BertI]

After much messing around with the Preview process such as trying to use setPreviewCallbackWithBuffer (thinking I might manually allocate one buffer at a time) I finally turned my attention to the jpeg conversion. Seems it is here where I hit the jackpot. Basically if I comment out the jpeg conversion line (and substitute a delay of ~150ms to try and keep the processing delay about the same as I found it to take to do the conversion) then no more memory leak.

I did also try something a bit more elaborate to try and discount possibility of repeated jpeg Dimming being a factor (I don't pretend to understand well yet how variables are managed memory wise). In the below jpeg2, picno, timenow are already Dimmed under Process Globals

Sub Camera1_Preview (PreviewPic() As Byte)
    If DateTime.Now > lastPreviewSaved + IntervalMs Then
        If picno<5 Then
            Dim jpeg() As Byte = camEx.PreviewImageToJpeg(PreviewPic, 70)
            jpeg2 = jpeg
            picno=picno+1
        Else
            Dim jpeg() As Byte = jpeg2
            timenow = DateTime.now
            Do While (DateTime.Now-timenow)<150
            Loop
        End If
        lastPreviewSaved = DateTime.Now
   
        CallSubDelayed2(Communicator, "Send", jpeg)
    End If
End Sub

With the above code no sign of a memory leak (at least in the few hours that I tested over - which would have resulted in several 10s of Mb before) but obviously only a single still repeatedly sent to the server app. Anyhow seems to support the theory that the leak is in the jpeg conversion process

Any ideas what might cause a memory leak in that sub? I can see various objects and output stream being Dimmed/initialized in there but my present understanding about the Androidy stuff beyond is pretty poor.

 

[JordiCP]

Just for curiosity

What happens if you don't call your "Send" routine (and do the PReviewImageToJpeg conversion normally)? Do you also have memory leaks?

 

[BertI]

I initially did think t was something to do with the communications (in my app) but after trying many different things without success I thought to test without communications and still got the same leak. So my next thought was the preview process itself and I thought I'd test with the CCTV example rather than my own app just to cut out other complexities. Again I still got the same kind of leak. Just as a double check I have now tried commenting out the 'send' routine in the CCTV example and I can already see the memory usage creeping up. So to me this is all pointing pretty strongly towards the jpeg conversion routine. However one thing that puzzles me is why the more powerful tablet with Android 4.4.4 appears to suffer less dramatically than the less well endowed and Android 4.4.2 tablet.

 

[Erel]

Comment out sections from PreviewImageToJpeg until the memory leaks are gone. This will help with finding the problematic point.

 

[BertI]

It wasn't quite so easy to comment bits out because of the dependencies of some objects on others but with a bit of moving some items to external declarations I did eventually narrow it down to this area:

    Dim out As OutputStream
    out.InitializeToBytesArray(100)
    r.RunMethod4("compressToJpeg", Array As Object(rect1, quality, out), _
        Array As String("android.graphics.Rect", "java.lang.int", "java.io.OutputStream"))
    Return out.ToBytesArray

I'm not sure how to separate out the use of the OutputStream but I did try at one point to flush and close it after the jpeg was delivered to the Main activity. Didn't appear to have any effect. Incidentally my original estimate for jpeg conversion time of 150ms was not quite right, infact some better positioned measurements show this to be more like 40ms for the same tablet and conditions.

Out of interest the leak rate has normally been around 36Mb/hour. After substituting a 40ms delay in place of the jpeg conversion above and just repeatedly returning a single stored image copy to the main activity (so that the rest would run as normally as possible) I have not even seen a 1Mb change over the last 24 hours.

 

[Erel]

I'm not sure how to separate out the use of the OutputStream but I did try at one point to flush and close it after the jpeg was delivered to the Main activity. Didn't appear to have any effect

I also suspected this however checking the source code reveals that the Close method of the memory stream doesn't do anything anyway.

This is the issue: https://code.google.com/p/android/issues/detail?id=71732

 

[BertI]

This is the issue: https://code.google.com/p/android/issues/detail?id=71732

Yes so looks like someone has already reported it, but presume no solution. I did also see the 'workaround' post which described going from YUV to RGB to JPEG but its somewhat beyond my skills/present knowledge to work out how to do this in B4A and there was also a comment about such a route significantly reducing frame processing rate though obviously can't verify. As a kind of groping in the dark I did try creating two services containing just the conversion part and then alternating between them at intervals of time (stop one service then start the other and vice versa) thinking vaguely that maybe the memory used by one would be cleared up when stopped. However this didn't work. I also tried using a piece of code which is supposed to invoke the garbage collector (found elsewhere in this forum) together with the service swapping but this didn't appear to work either. I guess you might say I've been clutching at straws to avoid the proper solution - which is somehow writing or encapsulating an alternative YUV-JPEG procedure. Ho hum

 

[xpectmore]

i have some issues too with cctv example.i am sending the frames to 192.168.x.x/test/put.php so i store the frame to mysql ...then i put it on screen with jquery.when i open first time the apk i see a green image on the website on pc browser. then i push phone's close key so the phone's screen is close then again i push the key close and i found open my apk and looking on the pc's screen browser i see a correct image not green anymore.what paintfull is this android.

if anyone needs anything to display feeds using html,css ,php,mysql ...


get.php

<?php //echo(base64_encode(file_get_contents("foto/pic.bmp")));



    $mysqli=mysqli_connect('localhost','root','','test');[ATTACH=full]53099[/ATTACH] [ATTACH=full]53100[/ATTACH]

    if (!$mysqli)
        die("Can't connect to MySQL: ".mysqli_connect_error());


    $stmt = $mysqli->prepare("SELECT pic from video ORDER BY id DESC LIMIT 1 ");


    $stmt->execute();
    $stmt->store_result();

    $stmt->bind_result($image);
    $stmt->fetch();
$mysqli->close();
    //header("Content-Type: image/bmp");
    //echo $image;
echo(base64_encode($image));
?>

index.php

<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <script src="js/jquery.js" type="text/javascript"></script>
    <style>
#r90 {
/*General*/
transform: rotate(90deg);
/*Firefox*/
-moz-transform: rotate(90deg);
/*Microsoft Internet Explorer*/
-ms-transform: rotate(90deg);
/*Chrome, Safari*/
-webkit-transform: rotate(90deg);
/*Opera*/
-o-transform: rotate(90deg);
/*alter opacity*/
/*opacity:0.4;
filter:alpha(opacity=40);*/
margin-top:50px;
}
    </style>
</head>
<body>
    <h1>Video test</h1>
    <div id="images"></div>

    <script type="text/javascript">
    function get_images(){
        $.ajax({
           url: "get.php",
           success: function(result)
           {
               img='<img id="r90" width="640px" height="480px" src="data:image/jpg;base64,'+result+'" />';
               $("#images").html(img);
           }, 
         });
    }
    $(document).ready(function(){
        get_images();
        setInterval(function(){get_images();}, 500);
    });

    </script>
</body>

<?php
$mysqli=mysqli_connect('localhost','root','','test');

    if (!$mysqli)
        die("Can't connect to MySQL: ".mysqli_connect_error());

    $stmt = $mysqli->prepare("INSERT INTO video (pic) VALUES(?)");
    $null = NULL;
    $stmt->bind_param("b", $null);

    //$stmt->send_long_data(0, file_get_contents("foto/pic.bmp"));
$stmt->send_long_data(0, file_get_contents("php://input"));

    $stmt->execute();

$mysqli->close();
?>

so remember in b4a cctv example to send frames to www.yourwebsite.ext/put.php
or 192.168.x.x/yourfolder/put.php
and you may create a folder "js" and then drop there an jquery.js else will not work

i have created in a database test the table video:

CREATE TABLE `video` ( `id` INT NULL AUTO_INCREMENT , `pic` LONGBLOB NOT NULL , PRIMARY KEY (`id`)) ENGINE = InnoDB;

if you look closely if any frame is sent to pc browser that's already rotated 90 degrees ...