B4J Question MQTT & Out of Memory

aaronk

Well-Known Member
Licensed User
Longtime User
Hi,

I am using the MQTT Broker in my B4J app and I am not 100% sure if the following error is released to the MQTT Broker & MQTT Client.

I have a MQTT Broker running in my B4J app. - https://www.b4x.com/android/forum/threads/mqttbroker.61548/#content

The B4J also connects to this broker. (MQTT Client).

My B4A/B4i apps will subscribe to topics and the B4J MQTT Client will publish to the topics that the MQTT in B4A/B4i are subscribed to.

I have a timer running in my B4J app which will run every 2 seconds to check if the MQTT Client is connected to the broker. (not sure if this is a good idea or not, or if this is the correct thing to do) If it's not connected then it will connect to the broker. This way the B4J app won't be disconnected from the broker and if it does it will be only for a short time.

For some reason the B4J app is running out of memory and I think it's related to the MQTT, since this is the last thing I added to the B4J app and now this issue is happening.

The Server is logging the following error..

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-21"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-23"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-24"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-0"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MQTT Ping: CloudServer1571683031905834035122"
Unexpected exception in the selector loop.
Unexpected exception in the selector loop.
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-1-thread-1"
An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
java.lang.OutOfMemoryError: Java heap space
message repeated 5 times: [ java.lang.OutOfMemoryError: Java heap space]
java.lang.OutOfMemoryError: Java heap space
message repeated 4 times: [ java.lang.OutOfMemoryError: Java heap space]
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp815992954-17905"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp815992954-17503"
java.lang.OutOfMemoryError: Java heap space
message repeated 2 times: [ java.lang.OutOfMemoryError: Java heap space]
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp815992954-18014"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp815992954-17056"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-22"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp815992954-16957"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp815992954-32"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Scheduler-1327536153"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "nioEventLoopGroup-2-1"
Unexpected exception in the selector loop.
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@2652bea7 in QueuedThreadPool[qtp1781071780]@6a28ffa4{STARTED,8<=8<=200,i=2,q=0}[ReservedThreadExecutor@1f61eb5b{s=0/2,p=0}]
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@2652bea7 in QueuedThreadPool[qtp1781071780]@6a28ffa4{STARTED,8<=8<=200,i=1,q=0}[ReservedThreadExecutor@1f61eb5b{s=1/2,p=0}]
Severe error during pipeline creation
An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
message repeated 2 times: [ An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:]
A task raised an exception.
An exception was thrown by a user handler's exceptionCaught() method while handling the following exception:
Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@2652bea7 in QueuedThreadPool[qtp1781071780]@6a28ffa4{STARTED,8<=9<=200,i=1,q=0}[ReservedThreadExecutor@1f61eb5b{s=1/2,p=0}]
Unexpected thread death: org.eclipse.jetty.util.thread.QueuedThreadPool$2@2652bea7 in QueuedThreadPool[qtp1781071780]@6a28ffa4{STARTED,8<=9<=200,i=0,q=0}[ReservedThreadExecutor@1f61eb5b{s=1/2,p=0}]
java.lang.OutOfMemoryError: Java heap space

In my server log I am seeing a lot of the following being logged:

Serious error processing the message org.eclipse.moquette.proto.messages.PublishMessage@2db4cb67 for session [clientID: CloudServer1571683031905834035122]org.eclipse.moquette.server.netty.NettyChannel@4b1dfbd2

The above doesn't seem to happen all the time.

The B4J app is the only device that will ever use the ClientID which starts with CloudServer. I am initialing it like:

B4X:
MqttClient.Initialize("MqttClient","tcp://127.0.0.1:5432","CloudServer" & DateTime.Now & Rnd(1000,9999999999999999))

So, I know it has a unique ClientID.

My B4J Broker is running version 1.03 and MQTT Client is running 1.00
I do notice there is a new MQTT Broker version available, but haven't updated to it yet.

As my server log is logging:
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MQTT Ping: CloudServer1571683031905834035122"

I am assuming the out of memory is because of the MQTT somehow ?

I am running my B4J app as service on the Linux VPS using:
B4X:
[Unit]
Description=CloudServer Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/java -jar /opt/CloudServer/CloudServer.jar
StandardOutput=syslog
StandardError=syslog

SuccessExitStatus=143
Restart=on-abort

[Install]
WantedBy=multi-user.target

Should I be running my B4J app by adding something to increase the Java Heap Space or is it due to the MQTT causing the error?

I am running the web server in my B4J app as well.

I can't seem to work out what is causing this error or how to fix it.
 
Last edited:

Erel

B4X founder
Staff member
Licensed User
Longtime User
- Worth updating to the latest version of MqttBroker. You just need to replace the libraries.
- You can run an external broker: https://mosquitto.org/download/
This way you will know for sure whether the problem is related to the broker or to a different component. This is also simple to do.
- Increasing the max heap can help unless there is a real memory leak.
 
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
- Worth updating to the latest version of MqttBroker. You just need to replace the libraries.
I updated to the new version today, so will see if that helps.

Increasing the max heap can help unless there is a real memory leak.
How do I do that? I am guessing I need to add something to the line ExecStart in the service that starts the B4J app (as per my service code above) ?
(might do that later, but want to prepare in advance so I know how to do it when I need to make this change)
 
Upvote 0

DonManfred

Expert
Licensed User
Longtime User
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
Looks like the MQTT Broker was the issue like I originally thought.

The new updated Broker (MQTTBroker2) didn't seem to fix the issue.

I then tried to increase the Java Heap Space to 2GB by using the following in my service to start my B4J App (I think this is correct below):

B4X:
[Unit]
Description=CloudServer Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/java -Xms2048m -Xmx2048m -jar /opt/CloudServer/CloudServer.jar
StandardOutput=syslog
StandardError=syslog

SuccessExitStatus=143
Restart=on-abort

[Install]
WantedBy=multi-user.target

This too didn't help, and the B4J app crashed within a day.

I then removed the MQTT Broker from my B4J app and installed an external one as suggested by Erel. (Mosquitto).
(It wasn't that hard to install)

It seems so far it's working fine. I had it running overnight and by B4J app is connecting to this external broker (running on the same VPS) and hasn't crashed since.

Either the MQTTBroker2 has a bug in it and causes it to crash, or the MQTTBroker2 has an issue running in my B4J app. I am using a HTTP server in my B4J as well. (Maybe it's clashing with it?)

Main thing is it looks like the external broker (Mosquitto) seems to have fixed my issue so far.
 
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
If it runs for awhile, then crashes, it almost sounds like a memory leak issue.
Since switching to Mosquitto everything seems to be working.

Looks like the MQTTBroker2 had the memory leak or had the issue.

So far it has been running for 1 Day and 12 hours and hasn't crashed, and my VPS is only using 486MB where when using the MQTTBroker2 it was using approx 700MB and increased to around 1.11GB before crashing. It used to crash within 3-4 hours. (My VPS has 2 CPU & 4GB of Memory allocated to it)

Seems that Mosquitto is the way to go if you want to use your own broker on your server. MQTTBroker2 might be OK for small low data projects.
 
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
Spoke too soon.

My B4J app was running for approx. 2 days and then crashed overnight.

The Server log shows:

Nov 1 00:40:52 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-23"
Nov 1 00:40:52 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-21"
Nov 1 00:48:32 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-0"
Nov 1 00:48:33 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-22"
Nov 1 00:48:48 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MQTT Ping: CloudServer15723376647161305457803"
Nov 1 00:48:57 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-24"
Nov 1 00:49:19 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MQTT Rec: CloudServer15723376647161305457803"
Nov 1 00:51:48 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-2"
Nov 1 01:20:30 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-3166"
Nov 1 01:33:33 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp920011586-3098"
Nov 1 01:40:44 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp920011586-19"
Nov 1 01:45:35 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-3257"
Nov 1 01:45:37 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-3260"
Nov 1 01:45:39 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-3259"
Nov 1 01:45:40 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-30"
Nov 1 01:45:40 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-3167"
Nov 1 02:08:17 java[944]: java.util.concurrent.TimeoutException: Idle timeout expired: 10800000/10800000 ms
Nov 1 02:08:18 java[944]: #011at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:166)
Nov 1 02:08:18 java[944]: #011at org.eclipse.jetty.io.IdleTimeout$1.run(IdleTimeout.java:50)
Nov 1 02:08:18 java[944]: #011at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
Nov 1 02:08:18 java[944]: #011at java.util.concurrent.FutureTask.run(FutureTask.java:266)
Nov 1 02:08:18 java[944]: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
Nov 1 02:08:18 java[944]: #011at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
Nov 1 02:08:18 java[944]: #011at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Nov 1 02:08:18 java[944]: #011at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Nov 1 02:08:18 java[944]: #011at java.lang.Thread.run(Thread.java:748)
Nov 1 02:30:33 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-28"
Nov 1 02:30:34 java[944]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp90205195-3210"

You can run an external broker: https://mosquitto.org/download/

I am now using the external Mosquitto (v3.1) on my VPS, but still crashes after a few days.

I am using the jMQTT version 1.00 for the B4J to connect to the Mosquitto broker so it can send the message. Couldn't find any updated version so assume it's the latest.

Running the MQTT Client Like:
B4X:
' following 2 lines are in the Process_Globals on the Main Module
    Public MqttClient As MqttClient                        ' jMQTT Lib
    Dim MQTT_Client_Options As MqttConnectOptions        ' jMQTT Lib
  
' the following lines are called as soon as my B4J app runs.
    MQTT_Client_Options.Initialize("username_here", "password_here")
    MqttClient.Initialize("MqttClient","tcp://127.0.0.1:8883","CloudServer" & DateTime.Now & Rnd(1000,9999999999999999))
    MqttClient.Connect2(MQTT_Client_Options)
  
    ' MQTT Broker is running on Port 8883 (non-SSL) as I change it from the default port.

My VPS has 4GB of RAM and 2 VCPU.

Around 3:30am is when I stopped and re-started my B4J app and everything come to life again.

I notice around 11PM - 11:30PM it started to increase in memory but don't notice anything in the server log that would cause the server to increase in memory

upload_2019-11-1_10-20-56.png


Something to point out, I am using the jServer in my B4J app which is hosting the HTTP pages and is using a WebSocket. All on different ports to each other. Could this cause any issues? I assume not since it's working for a few days before crashing and it would of crashed straight away if it was to blame ?

Everything seems to have been working fine (It was running for weeks/months without crashing) until I added the MQTT to my B4J app, so it seems that the MQTT Client has the issue somehow ?
 
Last edited:
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
You need to get a heap dump when you start seeing the out of memory errors
Great idea, and didn't realise this existed.

Just so I understand (had a look at the tutorial from the link) but just want to confirm, if I want it to automatically create it (I think option 3 from the tutorial) can I just run the following command now on my server, even knowing the Out of Memory is not happening and then it will create the file automatically when the issue happens? (I don't plan to stay up all night waiting for the error to happen)

B4X:
java -XX:+HeapDumpOnOutOfMemoryError

Or do I still need to run a command when the issue happens before I restart my B4J app ?
 
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
I am now running my app with a service like:

B4X:
[Unit]
Description=CloudServer Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/java -XX:+HeapDumpOnOutOfMemoryError -jar /opt/CloudServer/CloudServer.jar
StandardOutput=syslog
StandardError=syslog

SuccessExitStatus=143
Restart=on-abort

[Install]
WantedBy=multi-user.target

Fingers crossed this is correct, and will see what happens over the next few days and see what happens when it runs out of memory.
 
Upvote 0

DonManfred

Expert
Licensed User
Longtime User
I am guessing it creates the file once the out of memory issue happens
this is how i did interpreted the docs too.

Note that you only get one hprof file. after it is created you need to rename/move it to get another... As far as i understand the docs.
 
Upvote 0

aaronk

Well-Known Member
Licensed User
Longtime User
It crashed again. It didn't create any file :(

I looked in pretty much every folder on the server.

While the B4J app ran out of memory (before I started the app again) I ran the following command:

B4X:
jmap -dump:live,format=b,file=</opt/dump.hprof 14822

This will allow us to understand where is the memory leak.
Then it created a large dump file.
Here is a link to the dump file: https://drive.google.com/file/d/1O4jzPvXnmQaGepCxZu1MJIdtfAeeSVhv/view?usp=sharing

I have now made a change in the service and I am now starting the app like:
B4X:
[Unit]
Description=CloudServer Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/bin/java -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/dump.hprof -jar /opt/CloudServer/CloudServer.jar
StandardOutput=syslog
StandardError=syslog

SuccessExitStatus=143
Restart=on-abort

[Install]
WantedBy=multi-user.target

Fingers crossed this now creates the file when out of memory happens next time.
 
Upvote 0
Top