B4J Question AsyncStream COM-port buffer

peacemaker

Expert
Licensed User
Longtime User
HI, All

My app is scanning COM-ports, check the stream, looks for a data, receives the data, processes it....

And trying to catch the USB-COM device is disconnected so COM-port is suddenly lost, the connection is broken, but ... when the device is unplugged - the log shows data is being processed further during 5-7 seconds more before the AStream_Error event is fired.
But better to catch it ASAP.

What is it, big buffer ?
 

peacemaker

Expert
Licensed User
Longtime User
surprised that you get the Error event
Yes, it generates error:
Error: (NegativeArraySizeException) java.lang.NegativeArraySizeException: -1
java.lang.NegativeArraySizeException: -1
at jssc.SerialNativeInterface.readBytes(Native Method)
at jssc.SerialPort.readBytes(SerialPort.java:510)
at anywheresoftware.b4j.serial.Serial$1.read(Serial.java:165)
at anywheresoftware.b4j.serial.Serial$1.read(Serial.java:157)
at anywheresoftware.b4a.randomaccessfile.AsyncStreams$AIN.run(AsyncStreams.java:232)
at java.base/java.lang.Thread.run(Thread.java:834)
java.lang.RuntimeException: java.lang.NegativeArraySizeException: -1
at anywheresoftware.b4j.serial.Serial$1.read(Serial.java:172)
at anywheresoftware.b4j.serial.Serial$1.read(Serial.java:157)
at anywheresoftware.b4a.randomaccessfile.AsyncStreams$AIN.run(AsyncStreams.java:232)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.NegativeArraySizeException: -1
at jssc.SerialNativeInterface.readBytes(Native Method)
at jssc.SerialPort.readBytes(SerialPort.java:510)
at anywheresoftware.b4j.serial.Serial$1.read(Serial.java:165)
... 3 more
Error: (NegativeArraySizeException) java.lang.NegativeArraySizeException: -1
Connection is broken.


And when i'm trying to process the received buffer - all app is got very frozen, slow.
After disconnecting the COM-port - AStream_NewData is fired several seconds and all is stopped, and freezing is also out.

I do it in such class:

B4X:
Sub Class_Globals
    Dim mPort As String
    Dim mObject As Object
    Private sp As Serial
    Private const BAUDRATE As Int = 9600    '256000
    Private astream As AsyncStreams
    Dim LogMap As Map
    Dim mEmulation As Boolean, timEmul As Timer
    Dim emulation_sensor_id As Int
    Dim emulation_sensor_qty As Int = 1
End Sub

'Initializes the object. You can add parameters to this method if needed.
Public Sub Initialize (CallbackObject As Object, port As String, emul As Boolean, emul_id As Int) As Boolean
    mObject = CallbackObject
    mEmulation = emul
    LogMap.Initialize
    mPort = port
    emulation_sensor_id = emul_id
  
    If mEmulation Then
        timEmul.Initialize("timEmul", 1500)
        timEmul.Enabled = True
        Return True
    End If
  
    sp.Initialize("sp")
    Try
        sp.Open(port)
        sp.SetParams(BAUDRATE, 8, 1, 0)
        astream.Initialize(sp.GetInputStream, sp.GetOutputStream, "astream")
    Catch
        Dim e As String = port & " busy, passed by"
        Log(e)
        Main.toast.Show(e)
        CallSubDelayed2(mObject, "Show_Logs", e)
        Return False
    End Try
    Log(port & " подключен")
    others.AddToLog(Null, port & " connected")
    Return True
End Sub

private Sub AStream_NewData (Buffer() As Byte)
    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    LogMessage(s)
    others.Parse_RawData(s, mPort)
End Sub

private Sub AStream_Error
    Log("Error: " & LastException)
    Disconnect
    AStream_Terminated
End Sub

private Sub AStream_Terminated
    Log(mPort & ": " & "Connection is broken.")
    CallSubDelayed2(mObject, "Show_Logs", mPort & ": " & "Connection is broken.")
    others.AddToLog(Null, mPort & ": " & "Connection is broken.")
    CallSubDelayed2(mObject, "Broken_Port_Connection", mPort)
End Sub

Sub Disconnect
    astream.Close
    sp.Close
End Sub

private Sub LogMessage(Msg As String)
    CallSubDelayed2(mObject, "Show_Logs", mPort & ": " & others.AddToLog(LogMap, Msg))
End Sub

private Sub timEmul_Tick
    Dim l As List
    l.Initialize
  
    Dim emulated As String = "<sensor>" & emulation_sensor_id & " "
    l.Add(emulated)
    Dim k As Int = Rnd(1, 5)
    For i = 0 To emulation_sensor_qty - 1
        emulated = Chr(13) & Round2((Rnd(20000, 25000)/100/k).As(Float),2) & " " & Chr(10) & Chr(13) & Chr(10) & Chr(10)
        l.Add(emulated)
    Next
    emulated = "</sensor>"
    l.Add(emulated)
    For i = 0 To l.Size - 1
        others.Parse_RawData(l.Get(i), mPort)
    Next
    'others.Parse_RawData(emulated, mPort)
End Sub

Sub Send (txt As String)
    If txt.Length > 0 Then
        astream.Write(txt.GetBytes("utf8"))
    End If
End Sub
 
Last edited:
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
And this freezing is just without buffer processing !
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
I have replaced Java-11 sdk folder - and it works in Release mode better, but in debug - is also very freezing, strange.
 
Last edited:
Upvote 0

agraham

Expert
Licensed User
Longtime User
B4X:
private Sub AStream_NewData (Buffer() As Byte)
    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    LogMessage(s)
    others.Parse_RawData(s, mPort)
End Sub
This code is wrong. It is assuming that the first byte in the buffer is the start of a UTF8 character which, unless you are sending 7 bit ASCII is not a valid assumption. Serial comms cannot be relied upon to replicate complete packets of data as sent. What is sent as one string may well be received as several. Unless you can process each byte received independently you need a robust way of identifying the start and end of the data you are sending.

Reliable serial comms is harder than most people think if you are going to deal with disconnections and corrupted/missing characters.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
identifying the start and end
Sure, doing this way
B4X:
'buffer is Global map var
Sub Parse_RawData(s As String, port As String)
    If buffer.IsInitialized = False Then
        buffer.Initialize
    End If
 
    Dim hasStartMarker As Boolean = s.ToLowerCase.Contains(STARTMARKER)
    Dim hasStopMarker As Boolean = s.ToLowerCase.Contains(STOPMARKER)
    Dim BufferHasStartMarker As Boolean = buffer.Get(port).As(String).ToLowerCase.Contains(STARTMARKER)
    Dim BufferHasStopMarker As Boolean = buffer.Get(port).As(String).ToLowerCase.Contains(STOPMARKER)
    Dim buff As String = IIf(buffer.ContainsKey(port), buffer.Get(port), "")
 
    If hasStartMarker And BufferHasStartMarker = False Then
        Dim start As Int = s.ToLowerCase.IndexOf(STARTMARKER)
        s = s.SubString2(start, s.Length)
        buffer.Put(port, s)          'save start of the packet
        Return
    End If
 
    If hasStopMarker And BufferHasStopMarker = False And BufferHasStartMarker Then    'end part
        Dim Stop As Int = s.ToLowerCase.IndexOf(STOPMARKER)
        s = s.SubString2(0, Stop + STOPMARKER.Length)
        s = buff & s    'finish
    Else    'middle parts
        If BufferHasStartMarker And BufferHasStopMarker = False Then
            buffer.Put(port, buff & s)
            Return
        Else
            buffer.Remove(port)
            Sleep(0)
            Return
        End If
    End If

    buffer.Remove(port)

    If s.ToLowerCase.StartsWith(STARTMARKER) And s.ToLowerCase.EndsWith(STOPMARKER) Then
    Else
        'ignoring without markers
        Return
    End If
    s = s.Replace(STARTMARKER, "").Replace(STOPMARKER, "")
    s = Filter1(s)
    s = Filter2(s)
 
  Dim parts() As String = Regex.Split(" ", s)
  Process_Sensor(parts, port)    'save to db
End Sub

Sub Filter1(s As String) As String
    Dim s As String = s.Trim
 
    s = s.Replace(CRLF, " ")
    s = s.Replace(Chr(10), " ")
    s = s.Replace(Chr(13), " ")
    Return s
End Sub

Sub Filter2(s As String) As String
    s = Regex.Replace("\s+", s," ")
    Return s
End Sub

serial comms is harder than most people think
indeed !
 
Upvote 0

agraham

Expert
Licensed User
Longtime User
Sure, doing this way
You missed the point. Unless you are sending 7bit ASCII this line is a potential problem
B4X:
 Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
You should be parsing the raw byte data for stop/start patterns as you can't guarantee that transmission hasn't split a multi-byte UTF8 character. By the time your code gets to Parse_RawData it has potentially broken that data.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
But i guess only the very fist packet (at receiving start) can be broken.
And if i send only ASCII symbols for sure - i guess, no problem ? I did not catch for now.
Question is for strange freezing...
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
And the topic question about the system buffer - why after disconnecting - the data flow is going on several seconds more ? And how to prevent it immediately ?
 
Upvote 0

agraham

Expert
Licensed User
Longtime User
But i guess only the very fist packet (at receiving start) can be broken.
Wrong I'm afraid. Serial comms data will not arrive in the packet sizes as sent. You can expect partial packets of data at any time and will need to assemble them to find you complete packets of data.
why after disconnecting - the data flow is going on several seconds more
Windows serial comms drivers do buffer data, and who knows what the Java interface to them does as well. There is probably nothing you can do if the error arrives after all the buffered data has been received.
 
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
But if no way to setup the serial stream (buffer, or detect communication error state) in B4J (Java) to process immediately - how to be ?
 
Upvote 0

emexes

Expert
Licensed User
And the topic question about the system buffer - why after disconnecting - the data flow is going on several seconds more ? And how to prevent it immediately ?

My tack would be to back up a bit and comment out the processing, to see with more certainty what it actually coming in the serial port and when:

B4X:
private Sub AStream_NewData (Buffer() As Byte)
    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    LogMessage(DateTime.Now & " : " & s.Length & " = [" & s & "]")
    '''others.Parse_RawData(s, mPort)
End Sub

Also, are you sure it's UTF-8? Not ISO-8859-1 or windows-1252 or some similar 8-bit encoding? No binary data ie "raw" bytes?

If it's plain old boring 7-bit ASCII 🎶 there's no shame in that 🍻 just means one less thing to go wrong 👍
 
Last edited:
Upvote 0

peacemaker

Expert
Licensed User
Longtime User
what it actually coming
Here no any trouble, it's ASCII only for sure. But imagine that under Win10 this event AStream_NewData is going on to raise even after the device is unplugged and port is removed during 20-30 seconds. And only when whole this buffer is empty - _Error event is raised.
This is topic's question why and how to prevent.
 
Upvote 0
Top