walterf25

Expert
Licensed User
Longtime User
Hi Everyone, I am working on a project where I have a raspberry pi 4B where I have a b4j app running which communicates with a 32 channel Analog to Digital Converter we designed and built at work, the communication is via I2C protocol, this part works very well as far as I have been able to test.

The B4J app running on the raspberry pi collects around 1000 samples per second on each device, so far I have been testing with only 4 channels. I thought about using MTTQ library so that I can send the collected data of every iteration to another B4J App running on my Laptop PC so that I can plot the data on a XY line chart, while this is all working great, I have noticed that the Sub client_MessageArrived (Topic As String, Payload() As Byte) function is not raised as fast as I would expect.

I have made some time measurements on the app running on the raspberry pi and how long it takes to read between 32 bytes and 40 bytes and so far I have seen that it takes anywhere between 2 to 8 milliseconds per read, the data of all 4 channels is included in the 32 byte array read every iteration. The data is then sent or published via the MTTQ client library on every iteration.

What I would expect is that this data should arrive just as fast as it is sent, but I am seeing a delay of between 2 to 4 seconds on the PC laptop.

I was wondering if anyone has done any work using the MQTT library, I realize that using this library is probably not ideal for what I am trying to accomplish, essentially I am trying to plot data almost LIVE so that we can visually see changes on power, current and voltage on devices we are testing.

As context, we have also created a python script which does pretty much the same thing, it reads data from the i2c devices, saves data to a large array, then plotting the data. I see a huge difference between the speed using python and Java, however I would really love to find a better solution using B4X since I am more familiar with it than any other programming language.

Below is the relevant code from both b4j apps, the first one which reads data via i2c on the raspberry pi and publishes the data to a certain topic, and the second one which is running on my laptop pc which receives the data published to the same topic.

i2c_Read function:
Public Sub i2c1_read(register As Int, numofbytestoread As Int) As Byte()
    Dim byte_array(numofbytestoread) As Byte

    Dim successful As Boolean = False
    Do While successful = False
    Try
        Dim start As Long = DateTime.Now
    i2c1.ReadBytesI2C(register, byte_array)
        Dim stop As Long = DateTime.Now
        Log("time elapsed from i2c_read: " & ((stop - start)/1000))
    successful = True
    Catch
            successful = False
    End Try
    Loop

    Return byte_array
End Sub
This function is from the library I wrapped using the pi4j v2 java library, as you can see that's where I am taking the measurements of how long it takes to read 32 to 40 bytes of data and where I am seeing that it takes only between 2 to 8 milliseconds per read.

read data from i2c devices:
        For i = 0 To 999
        Dim start1 As Long = DateTime.Now
        Dim stop As Long = DateTime.Now
        i2c.SendRefresh(0x2)
        Dim vals() As Float = i2c.get_power_voltage_current

        xdata(i) = (stop - start) / 1000
        ydata(i) = vals(7)
        ydata2(i) = vals(4)
        ydata3(i) = vals(1)
        ydata4(i) = vals(10)
        Next
       
        mylist.Initialize
        mylist.Add(EncapsulateData(xdata, ydata))
        mylist.Add(EncapsulateData(xdata, ydata2))
        mylist.Add(EncapsulateData(xdata, ydata3))
        mylist.Add(EncapsulateData(xdata, ydata4))

        Dim b() As Byte = serializator.ConvertObjectToBytes(mylist)
        Try
            client.Publish2("datacollection", b, 2, False)
            Sleep(10)
        Catch
            Log("Error sending data")
        End Try
Here's where I collect data up to 1000 samples every iteration, then the data is sent via the MQTT library.

The following function is where the data published is received and sent to the XY charts to plot the data as is received.
message received function:
Sub client_MessageArrived (Topic As String, Payload() As Byte)
    Log("topic: " & Topic)
    Try
    Dim receivedObject As Object = serializator.ConvertBytesToObject(Payload)
    If Topic = "datacollection" Then
        Dim start As Long = DateTime.Now
        Dim m As mydata
        Dim m2, m3, m4 As mydata
        Dim l As List = receivedObject
        m = l.Get(0)
        m2 = l.Get(1)
        m3 = l.Get(2)
        m4 = l.Get(3)
        Dim xdata() As Object
        Dim ydata() As Object
        Dim xdata2() As Object
        Dim ydata2() As Object
        Dim xdata3() As Object
        Dim ydata3() As Object
        Dim xdata4() As Object
        Dim ydata4() As Object
        xdata = m.xxdata
        ydata = m.yydata
        xdata2 = m2.xxdata
        ydata2 = m2.yydata
        xdata3 = m3.xxdata
        ydata3 = m3.yydata
        xdata4 = m4.xxdata
        ydata4 = m4.yydata
            Log("xdata lenght: " & xdata.Length)
            Log("ydata lenght: " & ydata.Length)
       
        liveLC.updateLC("Board_ID", castObjectToDoubleArray(xdata), castObjectToDoubleArray(ydata))
        livelc2.updateLC("PPVUnreg_J10", castObjectToDoubleArray(xdata2), castObjectToDoubleArray(ydata2))
        liveLC3.updateLC("PPVUnreg_J14", castObjectToDoubleArray(xdata3), castObjectToDoubleArray(ydata3))
        liveLC4.updateLC("PPVUnreg_ProjRight", castObjectToDoubleArray(xdata4), castObjectToDoubleArray(ydata4))

        For ii = 0 To xdata.Length - 1
            Dim x As String = CastObjectToDouble(xdata(ii))
            Dim y As String = CastObjectToDouble(ydata(ii))
            Dim y2 As String = CastObjectToDouble(ydata2(ii))
            Dim y3 As String = CastObjectToDouble(ydata3(ii))
            Dim y4 As String = CastObjectToDouble(ydata4(ii))
            csvTable.Add(Array As String(x, y3, y2, y, y4))
        Next
        Dim stop As Long = DateTime.Now
        Log("time elapsed: " & (stop - start))
        ended = DateTime.Now
        Log("time elapsed betweem data received = " & (ended - begin))
    Else if Topic = "all/disconnect" Then
        Log("topic: " & Topic)
            Dim su As StringUtils
            su.SaveCSV(File.DirApp, "plot_data.csv", ",", csvTable)
    End If
    Catch
        Log("Error while processing messages")
        Dim su As StringUtils
        su.SaveCSV(File.DirApp, "plot_data.csv", ",", csvTable)
    End Try
End Sub

If anyone has any experience using the MQTT library and can provide some assistance in figuring out why the data is received every 2 to 4 seconds rather than a few milliseconds please let me know.
 

Attachments

  • Capture.PNG
    Capture.PNG
    261.5 KB · Views: 157

MicroDrie

Well-Known Member
Licensed User
Longtime User
With the example given, you only measure the time of 4 to 8 ms needed to read the time in your program because there is no program command between the start and stop:
Measure elapstime:
        Dim start1 As Long = DateTime.Now
        Dim stop As Long = DateTime.Now
        i2c.SendRefresh(0x2)
        Dim vals() As Float = i2c.get_power_voltage_current

        xdata(i) = (stop - start) / 1000
I think the best approach is, in addition to the total processing or lead time, to also find the one for each program step that needs the most processing time by measuring the start and stop time before and after each assignment. That gives you an idea of which of all processing tasks takes the most time. With that result you could see if it is possible and what the effect is when you try to make a part of the program faster.
 
Upvote 0

MicroDrie

Well-Known Member
Licensed User
Longtime User
I would like you to consider something else. You read in 1,000 samples. During reading, you perform some selection operations to further edit it after reading 1,000 samples. Why did you make that choice? Wouldn't it be better to separate reading and presentation by only storing the measured value and duration in an array that you will read out after the 1,000th or perhaps on the basis of an interrupt and process it separately from reading in? String operations in particular take a relatively long time and also lead to the need to clean up unused string values at certain times.
 
Upvote 0

Magma

Expert
Licensed User
Longtime User
Hi there...

in general MQTT "must" have a delay...... some delay... because MQTT using a middle (server/broker) to upload the data/message that stays there.... and then a second pc/client connects to broker to get the data message... so even using mqttt client and broker at the same pc you will have some delay...

Also the the length of message will also have a delay because need to serialize it and then send to broker... also when receiving need to convert it as object...

so if need faster way you need something different (no mqtt - but direct udp connection)... or faster broker...

For example you can use hivemq (that is very very very fast)...
 
Upvote 0

walterf25

Expert
Licensed User
Longtime User
More precisely: there is no program command between the start1 and stop

but even having spotted that, I'm still having trouble understanding how the timing is being done :oops:
So the total time is around 8 milliseconds, I only showed the time while the i2c read is happening to prove that is not the i2c library causing the issue.

With that said, the total time it takes from the command i send to read the data and the time after the data is received is only around 8 milliseconds which is still pretty fast for my needs. the time it takes to process the data and convert it to a serialized object only takes around 1 to 2 milliseconds. I have already done all this measurements so I know that is nothing either in the i2c library or the processing of the data that is causing this long delays.

i2c Data Collection:
    Do While connected = True
        For i = 0 To 999
        Dim start1 As Long = DateTime.Now
        Dim stop As Long = DateTime.Now
        i2c.SendRefresh(0x2)  ''' the refresh command is sent so that the ADC updates the data for each channel in a specific register
        Dim vals() As Float = i2c.get_power_voltage_current   ''this is the command that is sent to read data from the ADC, it retuns the data of all 4 channels at once
        Dim stop2 As Long = DateTime.Now
        Log("full difference: " & ((stop2 - start1)/1000))  '' here is where I get around 7 to 8 milliseconds in total
        xdata(i) = (stop - start) / 1000
        ydata(i) = vals(6)
        ydata2(i) = vals(4)
        ydata3(i) = vals(1)
        ydata4(i) = vals(10)
        Next
        
        mylist.Initialize
        mylist.Add(EncapsulateData(xdata, ydata))
        mylist.Add(EncapsulateData(xdata, ydata2))
        mylist.Add(EncapsulateData(xdata, ydata3))
        mylist.Add(EncapsulateData(xdata, ydata4))

        Dim b() As Byte = serializator.ConvertObjectToBytes(mylist)
        Try
            '''client.Publish2("datacollection", b, 2, False)
            client.Publish("datacollection", b) '' I really believe the issue is here, as I see the Error sending data pop up once in a while, but why???
            '''Sleep(100)
        Catch
            Log("Error sending data")
        End Try

    Loop
 
Upvote 0

walterf25

Expert
Licensed User
Longtime User
Hi there...

in general MQTT "must" have a delay...... some delay... because MQTT using a middle (server/broker) to upload the data/message that stays there.... and then a second pc/client connects to broker to get the data message... so even using mqttt client and broker at the same pc you will have some delay...

Also the the length of message will also have a delay because need to serialize it and then send to broker... also when receiving need to convert it as object...

so if need faster way you need something different (no mqtt - but direct udp connection)... or faster broker...

For example you can use hivemq (that is very very very fast)...
Thanks for your reply and your comments, I will definitely look into a different alternatives, as I mentioned I decided to try MQTT since it seemed easy to implement, while it was easy to set up I am seeing the long delay problems which is not good enough for what I am trying to do.

Thanks, I'll look into hivemq as well.

Walter
 
Upvote 0

techknight

Well-Known Member
Licensed User
Longtime User
You could look into RabbitMQ as well, its meant for a higher bandwidth usage. MQTT is a low bandwidth messaging protocol. I have used MQTT in high bandwidth environments before but it gets sketchy at best.
 
Upvote 0

walterf25

Expert
Licensed User
Longtime User
So a quick update, I was able to figure how to transfer and receive the data and display it almost live in the XY line chart.

The first approach I was doing was that I was sampling around 1000 samples per second and transferring the data at the end of each 1000 sample, this was indeed causing the huge delay I was seeing of around 2 to 3 seconds.

What I ended up dong is that I publish/transmit every sample as it is received, meaning I don't wait until I collect 1000 samples. This way I am seeing only a delay of around 7 milliseconds which coincides with the delay I am seeing when doing a i2c read of 30 bytes.

Data Collection:
Sub GetData
    Dim stop As Long
    Dim stop1 As Long
'''    Dim start As Long
'''    start = DateTime.Now
    Dim val As Double
    Dim val2 As Double
    Dim val3 As Double
    Dim val4 As Double
  
    Dim bc As ByteConverter
    For i = 0 To 999
        Dim start1 As Long = DateTime.Now
        Dim stop As Long = DateTime.Now
        i2c.SendRefresh(0x2)
        Dim vals() As Float = i2c.get_power_voltage_current

        xdata(i) = (stop - start) / 1000
        ydata(i) = vals(7)
        ydata2(i) = vals(4)
        ydata3(i) = vals(1)
        ydata4(i) = vals(10)
      
        mylist.Initialize
        mylist.Add(EncapsulateData2(xdata(i), ydata(i)))  ''this is needed because I was getting an error when trying to serialize array of floats
        mylist.Add(EncapsulateData2(xdata(i), ydata2(i)))
        mylist.Add(EncapsulateData2(xdata(i), ydata3(i)))
        mylist.Add(EncapsulateData2(xdata(i), ydata4(i)))

        Dim b() As Byte = serializator.ConvertObjectToBytes(mylist)
        Try
            client.Publish2("datacollection", b, 0, False)
        Catch
            Log("Error sending data")
        End Try
      
        stop1 = DateTime.Now
    Next

End Sub

This is the result
 

Attachments

  • daq.gif
    daq.gif
    477.8 KB · Views: 136
Last edited:
Upvote 0

aminoacid

Active Member
Licensed User
Longtime User
2-4 seconds is a very slow response. I would say the issue is with the Broker/Server you are using. I am sending lightning strike data close to 10msec apart at times but using my own Broker and never miss a packet. I suggest you setup your own local Broker if possible and try it. My recommendation is Mosquitto - very simple to setup and use.
 
Upvote 0
Top