Android Question MQTT Memory consumption

[Blueforcer]

In my current project, I have to send files via mqtt, for this I split them into several 5MB chunks and send them one after the other in a Do While loop as base64 in a json.
Everything works wonderfully. The only problem is that with each mqtt.publish the memory decreases by the size of the current chunk. Only when the file has been transferred completely the memory slowly become free again. But the problem is that if the file is too big I get an OutOfMemory exception at some point. QOS makes no difference here. I tested it by comment the mqtt.publish line, and the free memory stays stable.

Is there a problem with the garbage collector in the MQTT libary or is there a trick i can use to avoid the problem??

 

[emexes]

10MB chunks in 10s interval:

Waiting 10s
Mem: 374.222 MB
publish chunk #21
Waiting 10s
Mem: 360.644 MB
publish chunk #22
Waiting 10s
Mem: 347.076 MB
publish chunk #23
Waiting 10s
Mem: 347.08 MB
publish chunk #24
Waiting 10s
Mem: 317.041 MB
publish chunk #25
Waiting 10s
Mem: 302.8 MB
publish chunk #26
Waiting 10s
Mem: 302.467 MB
publish chunk #27
Waiting 10s
Mem: 289.032 MB
publish chunk #28
Waiting 10s
Mem: 275.456 MB
publish chunk #29
Waiting 10s
Mem: 275.447 MB
publish chunk #30
Waiting 10s
...

Brilliant!

But why aren't they evenly spaced? Why does the memory sometimes NOT decrease by the usual chunk size 13-14 MB? That makes me think that, during that 10 seconds, a chunk was added to the queue and also a chunk was removed from the queue.

Also mysterious - why did memory decrease by two chunks instead of the usual one chunk, from 347 MB to 317 MB?

 

[emexes]

I never said that im waiting for a ACK from the broker.

Nor did I. I said that I couldn't see any wait-until-confirmed loops.

You said:

The chunk is already received by the requesting client before the next one is sent

How can that be happening, without somebody or something waiting in-between for confirmation that a chunk has been received, before sending the next chunk? If it's not you, then it must be the MQTT library. Or it's not happening.

Your log above looks to me like it's not happening. 9 chunks were published, but memory usage only went down by 7 chunks, and the simplest explanation for that is that during that same time, MQTT managed to send 2 chunks over the network and then freed up their memory.

Please, please, please try the same test again, but with a 50 second sleep.

 

[Blueforcer]

Nor did I. I said that I couldn't see any wait-until-confirmed loops.

You said:



How can that be happening, without somebody or something waiting in-between for confirmation that a chunk has been received, before sending the next chunk? If it's not you, then it must be the MQTT library. Or it's not happening.

Your log above looks to me like it's not happening. 9 chunks were published, but memory usage only went down by 7 chunks, and the simplest explanation for that is that during that same time, MQTT managed to send 2 chunks over the network and then freed up their memory.

Please, please, please try the same test again, but with a 50 second sleep.

Because i can see it in the other client that the chunk was received, this takes about 2s.
It doesnt matter how long i should wait, in reality its simply too slow until the mqtt buffer is released.
I cant wait that much for each chunk.
I expect that the mem will be free as soon the message was sent, at the latest when the other client received it (with QOS 1-2), and not 20-30s later..

 

[emexes]

Because i can see it in the other client that the chunk was received, this takes about 2s.

Super! That sounds plausible over a 100 Mbps network that is operating efficiently.

What about if you transfer a 30-chunk file, as shown in the log? Can you see the chunks arrive in the other client about every 2 seconds?

Mem: 275.447 MB
publish chunk #30
Waiting 10s

Maybe the problem is the write speed to flash media by the other client.

I cant wait that much for each chunk.

We're in the diagnosis phase. Try with a 50 second Sleep. Do the chunks arrive in the other client abut 2 seconds after they're sent? At one chunk per 50 seconds, you've got time to check the chunk numbers too, ie make sure that it's the same chunk number that is received in the other client about 2 seconds after it's sent.

I expect that the mem will be free as soon the message was sent, at the latest when the other client received it (with QOS 1-2), and not 20-30s later..

I agree. That's what I'd expect too. But apparently that is not what's happening.

Before we can fix a problem, first we have to understand it.

I'm not proposing 50 second Sleeps as a fix; I am proposing them as a test to confirm that the memory is released after a chunk has been sent.

It could be that the MQTT system has a safety/quality feature than keeps messages for a minute or two aven after they've been sent, just in case there is a network fault and the hub/server/broker comes back with "hey, there was a problem with that last message, can you please send it again?" request. For most MQTT use cases, eg under 100 kB per minute, that wouldn't be a problem.

I'm not saying that *is* what MQTT does, I'm just brainstorming possibilities that could explain what you're seeing.

 

[emexes]

brainstorming possibilities that could explain what you're seeing

Maybe MQTT has metering/rationing/flow control that restricts average usage by a client/publisher to eg its total bandwidth divided by the number of clients it serves.

The first chunk/message goes through at full speed, because at that point your average usage of 0 is under whatever the allowance is, but when you send the second chunk and the broker sees your new average usage of 5 MB per second and says hey, slow down, 200 kB per second should be enough for any typical MQTT purpose... that could explain with what you're seeing.

 

[emexes]

Any luck in discovering reason for the unexpected high MQTT memory usage?

Another thought that occurred to me is:

if you are using a third-party broker/server/hub then perhaps there are service limits or caps or throttling being applied

or if your device is using a mobile broadband internet connection, perhaps it is happening there

(eg my parents moved to a wireless broadband internet connection for their home phone service - they have a monthly 20 GB at full speed, and then it is throttled to 2 Mbps ie 200 kB/second for the remainder of the month) (coincidentally similar to your logged 14 MB chunk per 45 seconds speed)