Android Question MJPEG Decoder artitecture

[goldmagnet]

Hello

I am trying to write a MJPEG stream receiver/decoder with B4A.

I have code that connects to the cameras, logs in, and recives the stream of data chunks, I am current displaying the data in a label.

The next part "Should" be simple and something I have coded on different platforms before. BUT I can not process the data in B4A. Normally I would use a string (the decoding is all string operations) and in the socket data received event:

Sub TcpStreams_NewData (buffer() As Byte)

BufferString = BufferString & BytesToString(buffer, 0, buffer.Length, "UTF8")

But this generates a huge memory leak, thats because the strings in B4A are imputable so a new string is used each time.

Next I tried using string builder and the append method, which generates the same memory issue, and now I am at a loss as to which structure I should use. This process is going to be run at least 25 times per second (thats how fast the label updates with the data), and perhaps 100 times per second depending on how the data is "chunked". Using a massive byte array, while maintaining a pointer might work but the string conversions would be too slow.

Can anyone help me out with some advice, I am new to B4A ?

Thanks

Huw

 

[kanaida]

Just curious, but whats the data bitrate?
Also why is text going onto a label?
Can the work be broken down into smaller pieces? I could have sworn this stuff worked in little frames of work.
Not sure if this would help, but BufferString =Null after each frame could release it. I'm not sure how that works as far as resources and the GC exactly but might help.

 

[goldmagnet]

Hi

I am copping the data into a label so I can "see" the data during development. The data includes jpeg "frames" which might be split over several buffer calls so I need to keep it somewhere in memory so I can work on the entire block of data to detect end and start markers etc. I have tried setting the bufferstring to NULL but it still eats memory, the only way to stop the memory issue is to set the string to null at the end of the routine not the end of the frame.

The bitrate will be dependent on the speed they can be received at, I am hoping for 25 fps with smaller resolutions and maybe half that for higher resolution images.

Thanks

Huw

 

[kanaida]

this sounds like the kind of thing that's built into the framework somewhere in a hardware accelerated fashion.
It might sound a little silly, but the media player might take a URL and presto
http://developer.android.com/guide/topics/media/mediaplayer.html

 

[goldmagnet]

A media play would not work, as we also need to implement our own protocol inside the MJPEG container as well, but need to do standard MJPEG first.

Our IOS app is explained here:

http://www.icode.co.uk/icatcher/pro/

I am just looking to see if we can achive this in B4A.

Thanks


Huw

 

[kanaida]

Gotcha. I know it's possible, there's ip camera clients that are functional on the market. Just used one the other day actually. I think it was called IP cam. There's a tool in the android development kit that lets you monitor memory usage/garbage collection etc. It might show some insight. It's called ddms.

http://developer.android.com/tools/help/index.html

As for doing it in b4a. I'm sure it's possible, but It's beyond my scope at the moment since I haven't had to deal with that much data in the same way. There's always the server option as well, a little live transcoding/processing. Do your hard work on the server and pass it some parameter or using a different port, then pass data that can be handled easily by the media player or otherwise. It may be a workaround for now.

 

[kanaida]

It's in the tools folder here for me:
C:\Program Files (x86)\Android\android-sdk\tools>

 

[Erel]

There isn't any memory leak in the code you provided. Note that Strings are immutable in Java as well.

It is however better to use StringBuilder to avoid creating many large string objects (which will be released by the garbage collector).

 

[goldmagnet]

Hello

Yes I read that about string builder yet the memory situation is the same if I use stringbuilder (see my first post in this thread), its behavior seems the same as using a string.

Huw

 

[goldmagnet]

Hi Erel

Yes I know that about Java, and yes I read the documentation on string builder, and as I said in the first post string builder generates the same memory issue in this case. I would like to know if you have any input on a method that wont result in huge memory use, it seems thats what string builder is for, it just does not work in the example I documented same huge memory use.

Thanks

Huw

 

[goldmagnet]

So I change the start of the routine to:

SBT.Append(BytesToString(buffer, 0, buffer.Length, "UTF8"))

BufferString = SBT.ToString

And at the end of the code I set:

BufferString = Null

And in the frame recivied code I have

SBT.Initalise

Still the same memory issue.

Huw

 

[Erel]

Which memory issue do you see? Are you getting an out of memory exception?

 

[goldmagnet]

No, and me saying its a memory issue could be misleading, that's just my best guess.

What happens is that the stream data is received and all appears to be working well, then over the course of the next 60 seconds the stream receiving gets slower and slower and eventually a message appears saying that the application has stopped working.

I notice that the 3.00 Beta mentioned increased performance in regard to HTTP operation "This update (together with an update to the Http library) allows you to send hundreds of requests at once." Do you think that this is relevant to my issue ?

Huw

 

[Erel]

I notice that the 3.00 Beta mentioned increased performance in regard to HTTP operation "This update (together with an update to the Http library) allows you to send hundreds of requests at once." Do you think that this is relevant to my issue ?

I'm pretty sure that it is not related.

Which error appears in the logs when the application crashes?

 

[goldmagnet]

Hi Erel

This is the error from the logs

main_tcpstreams_newdata (B4A line: 138)
BufferString = SBT.ToString
java.lang.OutOfMemoryError
at java.lang.String.<init>(String.java:432)
at java.lang.AbstractStringBuilder.toString(AbstractStringBuilder.java:642)
at java.lang.StringBuilder.toString(StringBuilder.java:663)
at anywheresoftware.b4a.keywords.StringBuilderWrapper.ToString(StringBuilderWrapper.java:57)
at b4a.example.main._tcpstreams_newdata(main.java:627)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:511)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:169)
at anywheresoftware.b4a.BA$3.run(BA.java:307)
at android.os.Handler.handleCallback(Handler.java:615)
at android.os.Handler.dispatchMessage(Handler.java:92)
at android.os.Looper.loop(Looper.java:137)
at android.app.ActivityThread.main(ActivityThread.java:4895)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:511)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:994)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:761)
at dalvik.system.NativeStart.main(Native Method)

 

[Erel]

Seems like you are creating a huge string.
Can you post the NewData code (that uses StringBuilder)?

 

[goldmagnet]

Hi Erel

I have stripped it right down.

This works fine:

Sub TcpStreams_NewData (buffer() As Byte)
     
    x=x+1
    SBT.Append(BytesToString(buffer, 0, buffer.Length, "UTF8")) 

    If x > 100 Then                                                                   
        SBT.Initialize
          Label1.Text = buffer.Length   
          x=0
    End If


End Sub

This crashes

Sub TcpStreams_NewData (buffer() As Byte)
     
    x=x+1
    SBT.Append(BytesToString(buffer, 0, buffer.Length, "UTF8")) 
    BufferString = SBT.ToString
    If x > 100 Then                                                                   
        SBT.Initialize
          Label1.Text = buffer.Length   
          x=0
    End If
 
    BufferString=Null

End Sub

The routine works, but gradually gets slower then crashes out.

I need to get at the data in a structure I can parse (the string builder in B4A has very limited functionality so I need access to the data outside string builder).

I then replaced the string with a byte array and populated that, only I run into the same issue. The byte array will populate but the routine to parse the data errors the same way.

Having been trying to do this for a month now I think I am going to have to go back to Java on this one, I can not find a method to parse a stream of MJPEG data that does not crash. Its almost like the code is running too fast for the garbage collection, but then our java MJPEG receiver works fine.

Huw

 

[Erel]

Basic4android StringBuilder is almost identical to Java StringBuilder.
The memory footprint is identical.

1. You are never clearing SBT. Eventually it will be too large to be kept in memory.
2. Are you receiving text data?

 

[goldmagnet]

Erel

Does a initalize not clear SBT ?

Any way I have also tried removing all the data (in place of iniatlize), and the same issue.

Huw

 

[Erel]

Yes. I missed that line. However are you receiving text data?

Add Log(SBT.Length) to this loop. I expect it too bee very large.