B4J Question Serial Port - asyncStream_NewData

[rgarnett1955]

Hi,

I am having trouble handling an asynch stream over a serial port. I cannot use prefixed streams because the external device Tiny Spectrum Analyser (TSA) does not provide this.


The process I am using is:

1. Send a "scan" command to the SA over the serial port AStream.write

2. Using AStream_NewData I process the stream recursively until I have received all of the data.

3. I then repeat the process with different scan parameters until I have scanned all bands

The amount of data returned by the TSA varies considerably and so sometimes a single return to AStream_NewData will have the entire response at other times two or three buffers will need to be processed.

To prevent the sending of new scan commands before all the received data has been received and processed, I am waiting for a resumable sub to be completed which I run when the processing of the data has been finalised. The TSA is half duplex, you cannot sends a command until the TSA has set its data with a "cr> ". If you do send a message before completion of the return datathe TSA stalls and must be power cycled to restart.

The problem I am having is the program is not waiting for the resumable sub and also the resumable sub is run without any call to it. I had a look at the B4X Language manual and noticed on page 69 at the bottom that:

Notes:
- It is safer to use CallSubDelayed than CallSub. CallSub will fail if the second sub is never paused
(for example if the sleep is only called based on some condition).
- There is an assumption here that FirstSub will not be called again until it is completed.

In particular the second point about the first sub not being called until it is completed is relavant in my case as it is quite possible for AStream_NewData to interrupt its own processing as new data becomes available.

A flow chart of the process I am using is shown below:

The code is:

'--------------------------------------------------------------------------------------------
'Scan each bands sequentially. The scan of a single band yields as reply 
'from the TSA for that band.
Public Sub scanAllBandsForThisSignalSet

    Dim bList As List
    bList.Initialize
    Main.progBandScan.Progress = 0
    Dim count As Int = 0
    bandIndex = 0
    
    Main.AllScansCompleteFlag = False
    
    bList = Main.Band_Data.MbandList
    
    For Each Record() As Object In bList
        count = count + 1
        
        'setBandSweepAndScan(band,      start,     stop,      BandMargin,  NoPoints,  rbw
        currentStart = Record(1) * 1e6
        CurrentStop  = Record(2) * 1e6
        setBandSweepAndScan(Record(0), Record(1), Record(2), Record(11),  Record(10), Main.Band_Data.SigSrcCurrentRecord.RBW_Khz)
        
        ' Wait for processing complete before sending another command
        Wait For (saDataReceived) Complete(done As Boolean)
        bandIndex = bandIndex + 1
    Next
    
    Main.AllScansCompleteFlag = True
    CallSubDelayed(Me, "saAllScansComplete")
End Sub

'--------------------------------------------------------------------------------------------
'This sub sends out a sweep command to sweep the points in a band
' It is a resumable sub and uses completion functions to signal to
' other application modules
Public Sub setBandSweepAndScan(band As Int, start As Double, _
    stop As Double, BandMargin As Double, NoPoints As Int, rbw As Int)
    'Construct the frequency strings
    Dim startStr As String
    Dim stopStr As String
    Main.scanCompleteFlag = False
    scanDataStreamText = ""
    noValidScanLines = 0
    Main.noScanBytesTotalExpected = networkPars.NO_CHARS_IN_SCAN_RESPONSE_PER_LINE * NoPoints + networkPars.NO_CHARS_IN_SCAN_RESPONSE_TERMINATION
    Main.noScanBytesReceived = 0
    
    bandNo = band
    
    start = start * 1e6
    If start < 1e6 Then
        startStr = NumberFormat(start, 1, 0) & "k"
    else if start < 1000e6 Then
        startStr = NumberFormat(start / 1000000, 1, 0) & "M"
    else if start > = 1 Then
        startStr = NumberFormat(start / 1e9, 1, 0) & "G"
    End If
    
    stop = stop * 1e6
    If stop < 1e6 Then
        stopStr = NumberFormat(stop, 1, 0) & "k"
    else if stop < 1000e6 Then
        stopStr = NumberFormat(stop / 1e6, 1, 0) & "M"
    else if stop > = 1 Then
        stopStr = NumberFormat(stop / 1e9, 1, 0) & "G"
    End If
    
    ' Sweep the band
    Main.dataReceivedFlag = False
    chCount = False
    dataStr = ""
'    noPointsThisScan = NoPoints
    cmd = "scan"
    commandStr = cmd & " " & startStr & " " & stopStr & " " & NoPoints & " " & 3
    noOfPackets = 0

    Main.writeCmdSerial(commandStr)
    Main.dataReceivedFlag = False
End Sub

Public Sub writeCmdSerial(cmdStr As String)
    Dim str As String
    str = cmdStr & Chr(13)
    byteBuffer = bc.StringToBytes(str, "UTF8")
    astream.Write(byteBuffer)
End Sub

'--------------------------------------------------------------------------------------------
Private Sub AStream_NewData (Buffer() As Byte)
    Dim s As String = BytesToString(Buffer, 0, Buffer.Length, "UTF8")
    SaCom.processCmdRx(Buffer)
End Sub

public Sub processCmdRx(Buffer() As Byte)
    replyError = False
    tempStr = BytesToString(Buffer, 0, Buffer.Length, "UTF8")

Select cmd
        Case "pause"
            checkReplyForErrorsShortCmds("pause")
            Main.dataReceivedFlag = True
        Case "lna"
            (...)
        Case "scan"   
            cleanUpScanData(Buffer)
            Main.dataReceivedFlag = True
    End Select
End Sub


'--------------------------------------------------------------------------------------------
Private Sub cleanUpScanData(buffer() As Byte)
    Log("Cleanup L429" & "   "  & DateTime.Time(DateTime.now))
    Dim newDataStart As Int = sb.Length
    sb.Append(BytesToString(buffer, 0, buffer.Length, charset))
    Dim s As String = sb.ToString
    Dim start As Int = 0
    For i = newDataStart To s.Length - 1
        noOfPackets = noOfPackets + 1
        Dim c As Char = s.CharAt(i)
        If i = 0 And c = Chr(10) Then '\n...
            start = 1 'might be a broken end of line character
            Continue
        End If
        If c = Chr(10) Then '\n
            Log("Cleanup chr(10)")
            CallSubDelayed2(Me, "checkAndProcessScanLine", s.SubString2(start, i))
'            checkAndProcessScanLine(s.SubString2(start, i))
            start = i + 1
        Else If c = Chr(13) Then '\r
            Log("Cleanup chr(13)")
            CallSubDelayed2(Me, "checkAndProcessScanLine", s.SubString2(start, i))
'            checkAndProcessScanLine(s.SubString2(start, i))
            If i < s.Length - 1 And s.CharAt(i + 1) = Chr(10) Then '\r\n
                i = i + 1
            End If
            start = i + 1
        End If
    Next
    If start > 0 Then sb.Remove(0, start)
End Sub

'--------------------------------------------------------------------------------------------
private Sub checkAndProcessScanLine(scanLineStr As String)
    If scanLineStr.Contains("level") Or scanLineStr.Contains("scan") Then
        Return
    End If
    
    scanDataStreamText = scanDataStreamText  & Chr(10) & scanLineStr
    noValidScanLines = noValidScanLines + 1
    
    If noValidScanLines = Main.Band_Data.getBandNoPointsArray(bandIndex) Then
        'Process the data   
        getScanListFromSaResult(scanDataStreamText)
        CallSubDelayed(Me, "saDataReceived")
    End If
    Return
End Sub

I guess one solution would be to use time-outs, however I wish to scan on a virtualy continuous basis with only a small "dead time" at the each of scan. As the amount end of the data stream can be determined, iI wish to use this as a flag for "all data received."

 

[agraham]

AStream_NewData to interrupt its own processing as new data becomes available.

This is incorrect. All events run on the main program thread so you will not suffer from re-entry problems.

 

[emexes]

I take it the serial communication is human-readable. Bonus!!

The TSA is half duplex, you cannot sends a command until the TSA has set its data with a "cr> "

What does "set its data" mean?

Do you mean that you wait for the TSA to send a cr (carriage return? ASCII 13) and a ">" prompt (and a space?) before sending a command?

What happens if you "press Enter" (ie send ASCII 13) to that prompt? Does it come up with an error, or just another prompt? Minor point, but it can be useful as a double-check to confirm that the TSA is alive and connected before sending the first command.

 

[emexes]

I guess one solution would be to use time-outs

Lol I was just about to mention that I've often found it simpler, easier and more bulletproof to just receive serial data until it stops for a specified period, or we know that the data has finished ie we've received a "cr>" prompt, or the expected amount of data.

I remember with the ELM OBDII device (15 years ago, yikes) the correct protocol was to wait for it to timeout and send a prompt, but the timeout was ~1 second, which was way too slow for graphing. The solution was that after we'd received the expected number of bytes for the requested PID, we'd "hit the Enter key anyway" which would short-circuit the timeout and immediately send back a prompt indicated it was ready for the next request.

 

[emexes]

Does TSA echo back the commands that you send it?

ie if you "type" the command "attenuate 17" & Chr(13)

are those 12 characters and 1 control character then also received back by AStream_NewData() ?

Also, is any TSA data in binary format ie not ASCII 0..127 ? More specifically, can it ever send data bytes with the high bit set? If so, then treating it as UTF-8 is going to screw that up, and it'd be better to keep the received data in its original Byte Array form until you know whether it's text or binary. Or maybe converting it from Byte to String "manually" using Chr(). Or use ISO-8859-1 instead of UTF-8 - it does a 1:1 mapping of Byte values 0..255 to Chr(0)..Chr(255).

 

[emexes]

I am jumping the gun here, but I'd probably have something like this:

Sub Process_Globals
    Dim SerialBuffer As String
    Dim SaPrompt As String = Chr(13) & ">"
End Sub

Private Sub AStream_NewData (Buffer() As Byte)
    SerialBuffer = SerialBuffer & BytesToString(Buffer, 0, Buffer.Length, "ISO-8859-1")    '1:1 mapping of Byte values to Char values
 
    Do While True
        Dim PromptAt as Int = SerialBuffer.IndexOf(SaPrompt)
        If PromptAt = 0 Then
            SaCom.HandlePrompt(SerialBuffer.SubString2(0, SaPromptLength))    'ie send next request
            SerialBuffer = SerialBuffer.SubString(SaPromptLength)
        Else If PromptAt > 0 then
            SaCom.HandleResponse(SerialBuffer.SubString2(0, PromptAt))
            SerialBuffer = SerialBuffer.SubString(PromptAt)
        Else
            Exit
        End If
    Loop
End Sub

 

[rgarnett1955]

Does TSA echo back the commands that you send it?

ie if you "type" the command "attenuate 17" & Chr(13)

are those 12 characters and 1 control character then also received back by AStream_NewData() ?

Also, is any TSA data in binary format ie not ASCII 0..127 ? More specifically, can it ever send data bytes with the high bit set? If so, then treating it as UTF-8 is going to screw that up, and it'd be better to keep the received data in its original Byte Array form until you know whether it's text or binary. Or maybe converting it from Byte to String "manually" using Chr(). Or use ISO-8859-1 instead of UTF-8 - it does a 1:1 mapping of Byte values 0..255 to Chr(0)..Chr(255).

Hi,

Thanks for your thoughts.

Q1 The TSA echos commands and finishes with a prompt "cr> " That is "c" "r" ">" space

Q2 The TSA sends everything back in ASCII with the exception of one command that I am not using. I am getting exactly what I expect from the TSA in terms of characters. All the characters (bytes) are being received. I have checked the stream data with a nice little program called: "Advanced Serial Pro" in spy mode. It picks up everything.

My problem isn't with the data it is with coordinating the write command with fragmented received data.

Its tricky because if I only request a small number of frequency points to make debugging easier, the fragmentation of the returned data doesn't occur. All the data is in the first call to AStream_NewData. What would be nice is for Astreams to have some sort of flow control with a definable buffer to ease the handling of longer data streams.

I think I may have found a solution using simple flags and a Do While loop viz:

        'Start a scan...'
        'Scanning initialisation stuff'   

        setBandSweepAndScan(Record(0), Record(1), Record(2), Record(11),  Record(10),...
 

        Do While scanCompleteSa = False
            Sleep(2)
        Loop

the scanCompleteSa variable is set by the data processing subs called from AStream_NewData.


So the scanCompleteSa variable is like a semaphore.


This seems to work, but I will have to do a lot of testing.

Kind regards
Rob

 

[rgarnett1955]

I take it the serial communication is human-readable. Bonus!!



What does "set its data" mean?

Do you mean that you wait for the TSA to send a cr (carriage return? ASCII 13) and a ">" prompt (and a space?) before sending a command?

What happens if you "press Enter" (ie send ASCII 13) to that prompt? Does it come up with an error, or just another prompt? Minor point, but it can be useful as a double-check to confirm that the TSA is alive and connected before sending the first command.

Sorry I meant that the TSA has received some data and has replied with its prompt "cr> "

The TSA command interface is designed to be used with a terminal program such as TerraTerm and so it echos back the characters entered up until the <LF>
It then returns a reply dependent on the command issued. At the end of the reply it sends a <LF>cr>space.


The lines of data are all of the form

"nnnnnnnnnn -n.nnnnnnnne-nn 000000000000\n"

I have no trouble pulling these out of the data using regex. I count the number of lines and compare them with the number of points scanned. These is one of these data records per point. The problem is that astream_NewData may only receive apart of the stream:

"nnnnnnnnnn -n.nnnnn"
"nnne-nn 000000\n"

However I may have figured it out using a kind of semaphore which I explained in my other reply.

 

[rgarnett1955]

This is incorrect. All events run on the main program thread so you will not suffer from re-entry problems.

Hi thanks for your reply,

but what is "This" specifically?

If I am executing asyncStream_NewData and some more data turns up what does the program do?

Does asyncStream_NewData interrupt itself or does it "queue up" waiting for a return from the asyncStream_NewData.

If there is a wait for sub in the asyncStream_NewData code doesn't that act like a return so that if new data has arrived whilst processing the previous data asyncStream_NewData will execute even though it hasn't returned?

I understand embedded systems using dma, deferred interrupts with semaphores/notification, but java is a total mystery to me.


Kind regards
Rob

 

[emexes]

Does the TSA echo back your commands and the terminating ASCII 13 (Enter) to you?

And do you think we can we rely on the prompt "cr> " character sequence not appearing by chance in the data responses?

 

[emexes]

pulling these out of the data using regex

I agree that the parsing of lines into fields could be done using regex, although it feels a bit like overkill for simple space-separated variables.

But if all the responses are in the form of LF (ASCII 10) terminated lines, then I think this layer is best done in the _NewData routine, after the latest data has been added to the buffer that is gathering together all the dribs and drabs of incoming data.

 

[emexes]

The end of a response, and return to a ready-for-next-command state, would be signified by a SerialBuffer ending in "cr> ".

Ideally it would have been preceded by a LF (ASCII 10) which would have extracted the last line of the response from SerialBuffer, leaving just the 4-character prompt.

When you switch the TSA on, presumably it transmits a "cr> " prompt.

But if you connect the TSA to the computer, after it has already transmitted the prompt, presumably you get nothing, and would rely on a no-activity timeout to send an exploratory Enter (ASCII 13) that would tickle the TSA into sending the prompt again. Is this the plan?

 

[agraham]

but what is "This" specifically?

Your statement/premise is incorrect.

but java is a total mystery to me.

This is not Java specific. Most modern OS like Windows, MacOS and Linux are event driven and programs are mainly single threaded.. A program idles monitoring a message queue waiting for messages to arrive that signal that an event has occurred. When one arrives it is processed by the appropriate Sub/method/function and when it completes returns to monitor the message queue. If a new event arrives while the program is dealing with another one the OS puts the message for the event in the message queue and the program will pick it up and action it when it returns to monitor the message queue so there are no re-entry problems to deal with.

Fragmented data is normal in all serial communications for all systems. You just have to deal with by buffering and identifying when whole packets of data have arrived.

 

[Chris2]

I must confess to not having read thoroughly the full thread here, but given that

The TSA sends everything back in ASCII with the exception of one command that I am not using

does this sound like AsyncStreamsText would do most of the job for you?

 

[rgarnett1955]

I must confess to not having read thoroughly the full thread here, but given that

does this sound like AsyncStreamsText would do most of the job for you?

Hi Chris

The cleanup sub is a variation of Erel's AsynchStreamsText and works fine.

The problem I am having is that the wait for subxxx does not appear to work. The program runs through the wait for without the associated subroutine being called.

I get the data OK, the problem is the program isn't waiting for the data to be complete before sending another scan command to the TSA.

 

[rgarnett1955]

Hi,

I'm still having problems with the serial port. Has anyone else got any ideas?

 

[emexes]

Don't use Wait For. Instead of the protocol being managed from above (ye olde programming style), push the bytes up from below instead (event-driven programming).

I'd start with a basic manual terminal conversation approach, and build up from from that.

 

[emexes]

Eg with these global and helper:

Sub Process_Globals
    Dim bc as ByteConverter
End Sub

Sub VisibleCodes(S As String) As String
    Dim Temp As String = S
    Temp = Temp.Replace(Chr(0), "{NUL}")
    Temp = Temp.Replace(Chr(8), "{BS}")
    Temp = Temp.Replace(Chr(9), "{TAB}")
    Temp = Temp.Replace(Chr(10), "{LF}")
    Temp = Temp.Replace(Chr(12), "{FF}")
    Temp = Temp.Replace(Chr(13), "{CR}")
    Temp = Temp.Replace(Chr(26), "{EOF}")
    Temp = Temp.Replace(Chr(27), "{ESC}")
    Temp = Temp.Replace(Chr(127), "{DEL}")
    Return Temp
End Sub

and these send and receive routines:

Sub SendCommand(str As String)
    byteBuffer = bc.StringToBytes(str, "ISO-8559-1")
    astream.Write(byteBuffer)
    Log(DateTime.Now & " tx: " & VisibleCodes(str))
End Sub

Sub astream_NewData (Buffer() As Byte)
    Dim s As String = bc.StringFromBytes(Buffer, "ISO-8559-1")
    Log(DateTime.Now & " rx: " & VisibleCodes(s))
End Sub

what does the log look like once you've connected up astream to the TSA and test it with eg:

SendCommand(Chr(13))
Sleep(2000)

SendCommand("version")
Sleep(1000)
SendCommand(Chr(13))
Sleep(2000)

SendCommand("version" & Chr(13))
Sleep(2000)

SendCommand("info" & Chr(13))
Sleep(2000)

SendCommand("vbat" & Chr(13))
Sleep(2000)