Android Question [felUSBSerial] Missing data while receiving bytes

Discussion in 'Android Questions' started by FredBerlin, Apr 28, 2019.

  1. FredBerlin

    FredBerlin Member Licensed User

    Hello everybody!
    After using felUSBSerial for some while, I realized that once a while some bytes are lost
    in the input stream. The procedure is as follows:
    Clicking the button "SEND" a short message is transmitted over an FTDI-cable to an external
    device. The device then answers with a byte message (simple ASCII chars) of a few hundred bytes.
    While receiving the continuous input stream (all bytes are chained together, i.e., after each stop bit the
    next start bit starts immediately) always some bytes are lost a random positions.
    Here is the code:

    Code:
    Sub Process_Globals
        
    Private usbserial As felUsbSerial
        
    Private manager As UsbManager
        
    Private bc As ByteConverter
    End Sub

    Sub Globals
        
    Private txtMessage As EditText
        
    Dim btnConnect As Button
        
    Dim btnClose As Button
        
    Dim btnSend As Button
    End Sub

    Sub Activity_Create(FirstTime As Boolean)
        
    Activity.LoadLayout("Test_felUsbSerial")
        
    If FirstTime Then
            manager.Initialize
        
    End If

        
    ' Dim btnConnect As Button
        btnConnect.Initialize("btnConnect")
        btnConnect.Text = 
    "Connect"
        
    Activity.AddView(btnConnect, 10dip10dip100dip40dip)

        
    ' Dim btnClose As Button
        btnClose.Initialize("btnClose")
        btnClose.Text = 
    "Close"
        
    Activity.AddView(btnClose, 120dip10dip100dip40dip)

        
    ' Dim btnSend As Button
        btnSend.Initialize("btnSend")
        btnSend.Text = 
    "Send"
        btnSend.Enabled = 
    False
        
    Activity.AddView(btnSend, 230dip10dip100dip40dip)
    End Sub

    Sub btnConnect_Click
        
    If manager.GetDevices.Length = 0 Then
            
    Log("No connected usb devices.")
            btnSend.Enabled = 
    False
        
    Else
            
    Dim device As UsbDevice = manager.GetDevices(0'assuming that there is exactly one device
            If manager.HasPermission(device) = False Then
                btnSend.Enabled = 
    False
                
    ToastMessageShow("Please allow connection and click again."True)
                manager.RequestPermission(device)
            
    Else
                
    usbserial.BUFFER_READ_SIZE = 16 * 1024 ' different settings make no difference
                usbserial.Initialize("RxD", device, -1)
                
    usbserial.BaudRate = 57600
                
    usbserial.DataBits = usbserial.DATA_BITS_8
                
    usbserial.StopBits = usbserial.STOP_BITS_1
                
    usbserial.Parity = usbserial.PARITY_NONE
                
    usbserial.FlowControl = usbserial.FLOW_CONTROL_OFF ' tested all settings here
                usbserial.StartReading
                btnSend.Enabled = 
    True
            
    End If
        
    End If
    End Sub

    Sub btnClose_Click
        
    If usbserial.IsInitialized Then
            
    usbserial.Close
            btnSend.Enabled = 
    False
        
    End If
    End Sub

    Sub btnSend_Click
        
    If usbserial.IsInitialized Then
            
    Dim s As String = "list" & Chr(13)
            
    Dim msg() As Byte = s.GetBytes("UTF8")
            txtMessage.Text = 
    ""
            
    usbserial.Write(msg)
        
    End If
    End Sub

    Private Sub RxD_DataAvailable (Buffer() As Byte)
        txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, 
    "UTF8")
        
    ' even if we don't add the new chars, but only show the last received ones, still there are lost chars,
        ' so seems like processing time of the txtMessage rendering doesn't seem to be the problem either
    End Sub
    Now the whole communication works properly using a standard terminal program on the same mobile
    device, i.e., many things can be ruled out: the missing bytes are not due to baud rate mismatch,
    half duplex switching, driver problems or the like.

    Any ideas?
    Best,
    Fred
     
  2. FredBerlin

    FredBerlin Member Licensed User

    ...just for the sake of completeness: there are no errors in debug/release mode while the application runs, transmits and receives data.
    Nonetheless some errors are reported during start-up, telling me felUSBSerial should be initialized early-on, but how could I initialize
    before I scan for the USB devices? I guess it is not related to the receive errors, but of course I may be wrong.
    M.

     
  3. JohnC

    JohnC Well-Known Member Licensed User

    See if you can reduce the baud rate down to something really low like 1200 baud (at both ends) and see if either the number of loss characters goes lower or that it no longer looses any characters.

    This could be an issue of data coming in too fast for the receiver to grab it all, and if slowing down the data makes things better, that would support this theory.

    And if the issue looks like it's due to overflow, then you will have to utilize either hardware or software flow-control at both ends.
     
    Last edited: Apr 28, 2019
  4. FredBerlin

    FredBerlin Member Licensed User

    Dear John,
    thank you for your suggestion.

    Unfortunately the external device only talks at 57600, so I cannot easily go down with the baud rate.
    But since "Serial USB Terminal" (PlayStore) can handle it easily (using the same mobile phone, hardware setting, USB device),
    it is clear that it is possible in principle.

    Within "Serial USB Terminal" messages 10 times as long and at even higher baud rates are received without lost characters
    - the TextEditBox even formats and scrolls up while receiving - so there must be plenty of leeway regarding processing power.

    The question remains, how to get there using B4A.
     
  5. agraham

    agraham Expert Licensed User

    It's a long shot but you say it is 'simple ASCII' but you are decoding UTF8. Are there any characters above ASCII 0x7F? If so this could cause a problem in UTF-8 decoding as characters over 0x7F are encoded as two bytes in UTF-8.

    To receive all the characters in a single byte code you probably need to specify a codepage such as
    Code:
    txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, "windows-1252")
    You can see the encodings supported on your system with ByteConverter.SupportedEncodings

    EDIT: I copied the wrong line of your code - now corrected !
     
    Last edited: Apr 28, 2019
  6. FredBerlin

    FredBerlin Member Licensed User

    Definitely all chars are below 0x7F.
    Also the dropped bytes are at different places each run, that rules out decoding problems.
    Would your theory be true, then the missing chars would be very systematic.
    Thank you for your suggestion!
    Fred
     
  7. FredBerlin

    FredBerlin Member Licensed User

    Although the receive buffer is large enough to comfortly hold the whole message,
    the "RxD_DataAvailable"-event is thrown several times, so the message is cut into
    a few pieces and glued together again using
    Code:
    txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, "UTF8")
    This is clear since if I instead use
    Code:
    txtMessage.Text = bc.StringFromBytes(Buffer, "UTF8")
    I only see the last piece of the message and this has random length.
    It appears to me, that the operating system periodically reports the received bytes so far,
    so the whole message is junked together randomly.

    Maybe the priority of the receiver-process of felUSBSerial is too low, so that once a while
    bytes get dropped?

    Best,
    Fred
     
  8. FredBerlin

    FredBerlin Member Licensed User

    The app where everything is working fine "Serial USB Terminal" from Kai Morich is coded in Java using Android Studio.

    There we have
    Code:
    interface SerialListener {
        void onSerialConnect      ();
        void onSerialConnectError (
    Exception e);
        void onSerialRead         (byte[] data);
        void onSerialIoError      (
    Exception e);
    }
    and
    Code:
    private void receive(byte[] data) {
            receiveText.append(new 
    String(data));
        
    }
    This appears to be equivalent to the B4A code I wrote, right?
     
  9. agraham

    agraham Expert Licensed User

    It's quite normal in serial comms for a message to be received as several individual pieces of varying length. This is why AsyncStreams exists. There should be no problem with a modern Android device in dealing with a continuous stream of data at 57600 baud. When I wrote the original UsbSerial library many years ago the Android devices then could happily cope at 115200 baud so a modern device should easily do so.

    I've done a lot of work with Android USB serial devices over the years and, bugs apart, never experienced randomly missing bytes so I am afraid that I am at a loss for further suggestions, apart from perhaps trying a different serial library.
     
  10. FredBerlin

    FredBerlin Member Licensed User

    Exactly. Should be no problem - 100% agreed.

    Still, I must be missing a detail, just don't see it.
     
  11. agraham

    agraham Expert Licensed User

    I assume you have tried this in release mode and the problem does persist then?
     
  12. FredBerlin

    FredBerlin Member Licensed User

    Yes, I tried release and debug mode, and all possible settings for hardware flow control. No difference.
     
  13. Erel

    Erel Administrator Staff Member Licensed User

    Better to monitor the input with:
    Code:
    Private Sub RxD_DataAvailable (Buffer() As Byte)
       
    Log("length: " & Buffer.Length)
       TotalLength = TotalLength + Buffer.Length 
    'global variable
       Log("Total: " & TotalLength)
    End Sub
    Is the length correct?
     
  14. FredBerlin

    FredBerlin Member Licensed User

    Dear Erel,
    the problem persists even when removing all costy operations and only adding
    two integers (TotalLength + Buffer.Length) the way you proposed. I added a Log("New_Try____")
    when the request button is pressed. The partial buffer lengths add up correctly, just obviously
    sometimes bytes are not received into Buffer() (and also not counted then, of course).
    Here is the logged output:

    The point being: Technically it must be possible, because the App "Serial USB Terminal" (PlayStore) never looses any byte.
    The only thing changed is the app used, everything else (hardware-wise) remains identical.

    Since it is a half-duplex connection (FTDI chip, which does the TXD/RXD switching internally) my guess is, that there
    could be an unwanted send/receive switching in-between the message chunks which would explain the loss of two bytes.
    Is there a method to get errors like "overrun error", "framing error" or the like, to see if there happened a receive error
    somewhere?

    Best,
    Fred
     
  15. agraham

    agraham Expert Licensed User

    As far as I recall all the USB serial chips that I have ever come across, including FTDI chips, are full-duplex internally and externally and can all support simultaneous operation in both directions.
     
  16. agraham

    agraham Expert Licensed User

    If you can, although it may not be possible as you say you cannot adjust the sending baud rate, I would try setting the sending device to add two stop bits. If this is not possible, as you are only dealing with ASCII set your receiver to seven data bits. Either of these should obviate the possibility of a framing error occurring.

    Depending on the FTDI chip in the device would also try my original UsbSerial library if it supports that chip as it uses different low level USB code to felUSBSerial and I know that it has worked at high baud rates for me in the past.
     
    mterveen and emexes like this.
  17. emexes

    emexes Well-Known Member Licensed User

    Do the dropped bytes usually/always occur at the boundary of/between the chunked data?
    Eg, where you logged:
    Code:
    New_Try________________________
    length: 
    87
    --> Total: 
    87
    length: 
    16
    --> Total: 
    103
    are the missing two bytes:
    - at the start of the 103 bytes (ie, first two bytes are dropped)
    - at the end of the 103 bytes (ie, last two bytes are dropped)
    - in between the 87 and 16 byte chunks?
    - within the 87 or 16 byte chunks?

    Also, is device always coming back with the *exact* *same* 105 bytes in response to the list request when you use the other serial terminal programs?

    Without seeing the output, I'm clutching at straws here, but: about once a decade I get tricked by something unexpected like tabs being expanded to spaces, or spaces being compressed to tabs.

    Certainly the original feel of the problem was that it was a bit format mismatch, and that sometimes a byte would cause a framing error and thus miss a byte or two. Now that I think about it, the fact that it misses either zero or two bytes, but seemingly not just one byte, hints that it could be a resynchronization issue.
     
  18. emexes

    emexes Well-Known Member Licensed User

    Or perhaps the serial device using XON/XOFF flow control (could happen irrespective of whether the serial-to-USB adapter is configured to use same).
     
  19. emexes

    emexes Well-Known Member Licensed User

    Another slightly radical idea: can you loop back the serial port ie connect RXD to TXD of the serial pins of the serial-to-USB adapter?

    Then send yourself packets of say 30-to-250 random bytes, make sure you receive them back shortly thereafter.

    Or can you generate test packets using another computer (laptop, even, with either a real serial port, or another USB-to-serial adapter). Or a different serial device - like, if you have an old modem hanging about, you could still have a "conversation" with it with AT commands, without needing to actually plug it into a phone line and make a call.
     
  20. FredBerlin

    FredBerlin Member Licensed User

    This goes to agraham and emexes:

    Dear agraham.
    Yes, full duplex on the USB-side, half duplex on the RS485. The sending device is out of my control, I have to live with it.
    Of course I could try to hook up test equipment to produce RS485 data stream to track it down, but that's kind of the last ressort.
    I startet off with your library, but the device is so new that it isn't supported. If you tell me how to trick your library into using the
    FTDI chip, I'd be happy to try out your library a second time. The post regarding the chip is here:

    https://www.b4x.com/android/forum/threads/usbserial-supported-devices.104260/#content

    Dear emexes.
    To check what has been received actually I'd need to process the data bytes (I did this in the beginning)
    but the latest tests as recommended by Erel throw away the data and just count the length. Maybe the next
    to look into is to write those byte into a queue and only later read them out and display them. This way
    timing issues are unlikely but I can still analyze where the missing bytes are.
    I also thought about hooking up a scope, but the problem doesn't seem to be on the hardware side,
    since other apps (or a PC connected) show everything just fine.
    Best idea I came up with to test next: Prepare a bunch auf independent byte-buffers and copy each received
    junk into one of them, so later on I have size + content for analysis.

    Best,
    Fred
     
Loading...
  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice