Android Question MJPEG Class, some issues using either raw socket or OKhttp implementations

[BertI]

I was testing the MJPEG decoder example from here: https://www.b4x.com/android/forum/threads/b4x-class-mjpeg-decoder.73702/post-468057 on an Android tablet whose SDK version shows up as 22. The example from that link was ported to B4A 8.50. The video showed up ok when I tested on some random mjpg camera. However there was an odd symptom a little while after exiting the application:

** Activity (main) Pause, UserClosed = true **
sending message to waiting queue (astream_newdata)
sending message to waiting queue (astream_newdata)
sending message to waiting queue (astream_newdata)
...
...
sending message to waiting queue (astream_newdata)
sending message to waiting queue (astream_newdata)
Ignoring event (too many queued events: astream_newdata)
sending message to waiting queue (astream_newdata)
Ignoring event (too many queued events: astream_newdata)
sending message to waiting queue (astream_newdata)

I added an extra line of code to log each time astream_newdata sub was entered and it seems that the activity pause event did not trigger for quite a while after I actually exited the application (at least according to data appearing in log) and even after that point something seems to be going on.

I then found this implementation using OKhttp instead of raw sockets: https://www.b4x.com/android/forum/threads/consuming-http-streams.111542/post-695615

Again ported to B4A 8.50. Same kind of symptom but obviously different target, eg.:

sending message to waiting queue (data_available)
Ignoring event (too many queued events: data_available)

Another interesting thing I noted is that the image update rate in the B4A implementation was quite poor, e.g sometimes a few seconds before a new frame appeared (the video camera handily had a time label), whereas in the B4J implementation the streaming looked smooth. Also no such after exit queue symptoms for the b4J implementations. Suppose frame slowness aspect could also be some function of the Android tablet I was testing with, but I do know that using a webview on the tablet it does stream ok. Problem with webview I gather is that not all support mjpeg decoding, also I'm finding it difficult to consistently control the image scaling for webview (https://www.b4x.com/android/forum/threads/webview-retain-zoom-setting.161923/post-993118)

The ported version of the OKhttp version is attached.

 

[drgottjr]

even if/when you close the connection, there are still data
in the pipeline running on a separate thread. since the connection is async, those data
keep accumulating, waiting to be processed by your app, until
the close connection command is handled. if you do not close
the connection, it could keep running. erel states in the
initial mpeg library posting that the developer has to handle
closing the connection. my guess would be that you did not do
this in your original project.

in the okhttputils2 version, a connection close option was
provided, and you did you use it. i ran your example on a
pixel3a. i only received 2 log messages relating to data still
in the queue. if you're receiving more message about a waiting
queue, i would attribute that to the device's inability to handle
receiving the data at a faster rate.

you could argue that when the connection is closed, any
lingering data should be dumped. it probably could be
implemented one way or the other at the developer's discretion.
it is unlikely to be handled at the application level. once
async transfer is chosen, however, you effectively limit some
other options. it's tough to have things both ways.

 

[BertI]

Thanks for that feedback. For the original raw socket version I did try to use what I thought might close off the connection - like Astream.close and sock.close on Activity_pause and even on the AStreams_Error and Terminated events (though latter probably doesn't make sense). But I couldn't see any other relevant method I might use to force the queuing to stop. I suppose as you describe it being a separate thread this might mean something to somehow stop that thread. Also I don't yet know whether this behaviour actually represents a problem beyond the fact that it is showing up in the log. The main reason I was investigating this aspect is because I found the application itself grinding almost to a halt and unresponsive in other areas such as responding to button clicks or even to the standard Android navigation bar. For example I'd often have to hold my finger on the Android < button for several seconds before it allowed me to exit by that route.

I wondered whether handling the 1Mb Data array was perhaps taking up too much resource so I reduced this to 100K (image frames being around 50K in this case). However this did not make much/any difference. I know the tablet device itself is not particularly fast but if I use a webview on the same device to access the same mjpg stream as in the example it runs smoothly and frame rate is good.

 

[OliverA]

However there was an odd symptom a little while after exiting the application

It looks like you also needed to release the Response object. This still will leave you with a "Stream Finish" waiting queue entry, but that's because the Activity is in the background and paused by the time that event is raised. This issue is resolved if you switch the project to a B4XPages-based project.

I attached "solutions" in both Activit-based and B4XPages-based versions. Note, for the Activity-based version B4XBytesBuilder is part of the B4XCollections library and for the B4XPages-based version, B4XBytesBuilder is part of the B4XPages library. Also, the package name of your app (for initializing the inline Java class) can be derived via the Application.PackagName property.

Another interesting thing I noted is that the image update rate in the B4A implementation was quite poor, e.g sometimes a few seconds before a new frame appeared (the video camera handily had a time label)

I don't see this on a Galaxy S4, running Android 5.01

 

[BertI]

What can I say? The generosity of spirit in this community is perhaps one example of why humankind may yet have hope.

Must admit to still being scared of B4XPages , though I do have B4A version 12.80 installed for exercising myself on when I manage to get islands of time.. So I did manage to compile and install your B4X rendition. As you say this did resolve the queuing symptom, however the image frame update rate was still much the same as for the Activity based version. I also tried your Activity based version with the response release but the queuing symptom still showed up with that. Eventually I got hold of a different device to try things with and the issues pretty much disappeared. So I think this points to the ability of the device to process the computations in good time as being the exacerbating factor.

I imagine a webview is doing pretty much the same work under the hood to extract and display the jpg frames but probably less affected by this slow device because its code is more efficient in some way.