B4J Question Ideas for MQTT client

[npsonic]

I have multiple MQTT clients around the world. I'm hosting MQTT broker on dedicated server for these clients.
There is also one MQTT client app on same dedicated server that is hosting MQTT broker. This client receives messages from every other MQTT client and stores the data into database.

So MQTT clients every where are sending data every 10 seconds to there specified rooms and one MQTT client on dedicated server is listening all these rooms and gathering data.

Why this kind of syncing service is created with MQTT and not with web server and http post methods is beyond me.


Now client app on server is just processing every message in order as they came, but problem is that more there is outside clients sending data the bigger is list growing that is used as an buffer for arriving messages.
Processing data takes time as it is in json that has to be parsed before inserting into database.
Should I somehow create a new thread for each message to counter problem of growing buffer?

 

[DonManfred]

Should I somehow create a new thread for each message to counter problem of growing buffer?

i would use a temporary table in a sqlite db. Each message coming in is stored in the db as raw-data... the json like it comes with the message.

using a timer i would check new content in the db and process the data, converting the json to real database entries and then remove the imported entry from the temp-table.

 

[MarkusR]

maybe save the incoming data at server to files and then let multiple apps get one file of this json message for process it.

 

[npsonic]

i would use a temporary table in a sqlite db. Each message coming in is stored in the db as raw-data... the json like it comes with the message.

using a timer i would check new content in the db and process the data, converting the json to real database entries and then remove the imported entry from the temp-table.

maybe save the incoming data at server to files and then let multiple apps get one file of this json message for process it.

SQLite actually supports multiple connections from different threads, so if I would use one database where I store messages as they come and multiple threads to process messages this might work.

 

[BillMeyer]

I have this type of setup. It works perfectly for me. Only difference is I write to a MySQL database.

You say you have a dedicated server (Mine has 2x Xeon Processors running at 3.2gHz with 16gb Ram - Server Windows 2016) that runs only your Broker and a Client to parse and save the incoming data. Well, depending on the amount of data you send per message, you are going to have a hard time trying to strain a server that has any type of substance about it.

On my server I run CKVS, MySQL, Mosquitto Server and a Client for the MQTT messages and I have tried to throw the book at it and see what happens and so far I have only managed to get to a 6% load.

The reason I used MQTT is that using HTTP and a Web Service is that there is a "layer" between the client and the server, where as with MQTT it is straight to the broker.

What are you sending in the MQTT Messages ?

 

[npsonic]

I have this type of setup. It works perfectly for me. Only difference is I write to a MySQL database.

You say you have a dedicated server (Mine has 2x Xeon Processors running at 3.2gHz with 16gb Ram - Server Windows 2016) that runs only your Broker and a Client to parse and save the incoming data. Well, depending on the amount of data you send per message, you are going to have a hard time trying to strain a server that has any type of substance about it.

On my server I run CKVS, MySQL, Mosquitto Server and a Client for the MQTT messages and I have tried to throw the book at it and see what happens and so far I have only managed to get to a 6% load.

The reason I used MQTT is that using HTTP and a Web Service is that there is a "layer" between the client and the server, where as with MQTT it is straight to the broker.

What are you sending in the MQTT Messages ?

There is also DNN website hosted on that server, but it isn't even close to as powerful as your server. Utilization is constantly somewhere at 40%.

It's basically all for syncing client databases to database on server. Problem is that system that we are talking about generates about ~5 million rows of complex data per client in each year.
Data stored at server is visible for the clients through website hosted on that same server.

When server is starting to hit about ~80% load constantly it's probably time to upgrade, but not yet.