Android Question [felUSBSerial] Missing data while receiving bytes

[FredBerlin]

Hello everybody!
After using felUSBSerial for some while, I realized that once a while some bytes are lost
in the input stream. The procedure is as follows:
Clicking the button "SEND" a short message is transmitted over an FTDI-cable to an external
device. The device then answers with a byte message (simple ASCII chars) of a few hundred bytes.
While receiving the continuous input stream (all bytes are chained together, i.e., after each stop bit the
next start bit starts immediately) always some bytes are lost a random positions.
Here is the code:

Sub Process_Globals
    Private usbserial As felUsbSerial
    Private manager As UsbManager
    Private bc As ByteConverter
End Sub

Sub Globals
    Private txtMessage As EditText
    Dim btnConnect As Button
    Dim btnClose As Button
    Dim btnSend As Button
End Sub

Sub Activity_Create(FirstTime As Boolean)
    Activity.LoadLayout("Test_felUsbSerial")
    If FirstTime Then
        manager.Initialize
    End If

    ' Dim btnConnect As Button
    btnConnect.Initialize("btnConnect")
    btnConnect.Text = "Connect"
    Activity.AddView(btnConnect, 10dip, 10dip, 100dip, 40dip)

    ' Dim btnClose As Button
    btnClose.Initialize("btnClose")
    btnClose.Text = "Close"
    Activity.AddView(btnClose, 120dip, 10dip, 100dip, 40dip)

    ' Dim btnSend As Button
    btnSend.Initialize("btnSend")
    btnSend.Text = "Send"
    btnSend.Enabled = False
    Activity.AddView(btnSend, 230dip, 10dip, 100dip, 40dip)
End Sub

Sub btnConnect_Click
    If manager.GetDevices.Length = 0 Then
        Log("No connected usb devices.")
        btnSend.Enabled = False
    Else
        Dim device As UsbDevice = manager.GetDevices(0) 'assuming that there is exactly one device
        If manager.HasPermission(device) = False Then
            btnSend.Enabled = False
            ToastMessageShow("Please allow connection and click again.", True)
            manager.RequestPermission(device)
        Else
            usbserial.BUFFER_READ_SIZE = 16 * 1024 ' different settings make no difference
            usbserial.Initialize("RxD", device, -1)
            usbserial.BaudRate = 57600
            usbserial.DataBits = usbserial.DATA_BITS_8
            usbserial.StopBits = usbserial.STOP_BITS_1
            usbserial.Parity = usbserial.PARITY_NONE
            usbserial.FlowControl = usbserial.FLOW_CONTROL_OFF ' tested all settings here
            usbserial.StartReading
            btnSend.Enabled = True
        End If
    End If
End Sub

Sub btnClose_Click
    If usbserial.IsInitialized Then
        usbserial.Close
        btnSend.Enabled = False
    End If
End Sub

Sub btnSend_Click
    If usbserial.IsInitialized Then
        Dim s As String = "list" & Chr(13)
        Dim msg() As Byte = s.GetBytes("UTF8")
        txtMessage.Text = ""
        usbserial.Write(msg)
    End If
End Sub

Private Sub RxD_DataAvailable (Buffer() As Byte)
    txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, "UTF8")
    ' even if we don't add the new chars, but only show the last received ones, still there are lost chars,
    ' so seems like processing time of the txtMessage rendering doesn't seem to be the problem either
End Sub

Now the whole communication works properly using a standard terminal program on the same mobile
device, i.e., many things can be ruled out: the missing bytes are not due to baud rate mismatch,
half duplex switching, driver problems or the like.

Any ideas?
Best,
Fred

 

[FredBerlin]

...just for the sake of completeness: there are no errors in debug/release mode while the application runs, transmits and receives data.
Nonetheless some errors are reported during start-up, telling me felUSBSerial should be initialized early-on, but how could I initialize
before I scan for the USB devices? I guess it is not related to the receive errors, but of course I may be wrong.
M.

Logger verbunden mit: samsung SM-G975F
--------- beginning of crash
--------- beginning of main
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
Buffer size: 16384
Buffer size: 16384
** Activity (main) Pause, UserClosed = true **
** Service (starter) Destroy (ignored)**
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
java.lang.RuntimeException: Unable to create service b4a.example.starter: java.lang.RuntimeException: java.lang.NullPointerException: Attempt to invoke virtual method 'void java.iutputStream.write(byte[])' on a null object reference
at android.app.ActivityThread.handleCreateService(ActivityThread.java:3761)
at android.app.ActivityThread.access$1400(ActivityThread.java:236)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1800)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loop(Looper.java:214)
at android.app.ActivityThread.main(ActivityThread.java:7032)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:494)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:965)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException: Attempt to invoke virtual method 'void java.iutputStream.write(byte[])' on a null object reference
at anywheresoftware.b4a.shell.Shell.virtualAssets(Shell.java:164)
at anywheresoftware.b4a.shell.Shell.start(Shell.java:102)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:92)
at b4a.example.starter.onCreate(starter.java:34)
at android.app.ActivityThread.handleCreateService(ActivityThread.java:3749)
... 8 more
Caused by: java.lang.NullPointerException: Attempt to invoke virtual method 'void java.iutputStream.write(byte[])' on a null object reference
at anywheresoftware.b4a.shell.ShellConnector.sendControlMessage(ShellConnector.java:61)
at anywheresoftware.b4a.shell.Shell.virtualAssets(Shell.java:124)
... 12 more
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Service (starter) Destroy (ignored)**
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = false **
** Activity (main) Resume **
Buffer size: 16384
** Activity (main) Pause, UserClosed = true **

 

[JohnC]

See if you can reduce the baud rate down to something really low like 1200 baud (at both ends) and see if either the number of loss characters goes lower or that it no longer looses any characters.

This could be an issue of data coming in too fast for the receiver to grab it all, and if slowing down the data makes things better, that would support this theory.

And if the issue looks like it's due to overflow, then you will have to utilize either hardware or software flow-control at both ends.

 

[FredBerlin]

Dear John,
thank you for your suggestion.

Unfortunately the external device only talks at 57600, so I cannot easily go down with the baud rate.
But since "Serial USB Terminal" (PlayStore) can handle it easily (using the same mobile phone, hardware setting, USB device),
it is clear that it is possible in principle.

Within "Serial USB Terminal" messages 10 times as long and at even higher baud rates are received without lost characters
- the TextEditBox even formats and scrolls up while receiving - so there must be plenty of leeway regarding processing power.

The question remains, how to get there using B4A.

 

[agraham]

It's a long shot but you say it is 'simple ASCII' but you are decoding UTF8. Are there any characters above ASCII 0x7F? If so this could cause a problem in UTF-8 decoding as characters over 0x7F are encoded as two bytes in UTF-8.

To receive all the characters in a single byte code you probably need to specify a codepage such as

 txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, "windows-1252")

You can see the encodings supported on your system with ByteConverter.SupportedEncodings

EDIT: I copied the wrong line of your code - now corrected !

 

[FredBerlin]

Definitely all chars are below 0x7F.
Also the dropped bytes are at different places each run, that rules out decoding problems.
Would your theory be true, then the missing chars would be very systematic.
Thank you for your suggestion!
Fred

 

[FredBerlin]

Although the receive buffer is large enough to comfortly hold the whole message,
the "RxD_DataAvailable"-event is thrown several times, so the message is cut into
a few pieces and glued together again using

txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, "UTF8")

This is clear since if I instead use

txtMessage.Text = bc.StringFromBytes(Buffer, "UTF8")

I only see the last piece of the message and this has random length.
It appears to me, that the operating system periodically reports the received bytes so far,
so the whole message is junked together randomly.

Maybe the priority of the receiver-process of felUSBSerial is too low, so that once a while
bytes get dropped?

Best,
Fred

 

[FredBerlin]

The app where everything is working fine "Serial USB Terminal" from Kai Morich is coded in Java using Android Studio.

There we have

interface SerialListener {
    void onSerialConnect      ();
    void onSerialConnectError (Exception e);
    void onSerialRead         (byte[] data);
    void onSerialIoError      (Exception e);
}

and

    private void receive(byte[] data) {
        receiveText.append(new String(data));
    }

This appears to be equivalent to the B4A code I wrote, right?

 

[agraham]

It's quite normal in serial comms for a message to be received as several individual pieces of varying length. This is why AsyncStreams exists. There should be no problem with a modern Android device in dealing with a continuous stream of data at 57600 baud. When I wrote the original UsbSerial library many years ago the Android devices then could happily cope at 115200 baud so a modern device should easily do so.

I've done a lot of work with Android USB serial devices over the years and, bugs apart, never experienced randomly missing bytes so I am afraid that I am at a loss for further suggestions, apart from perhaps trying a different serial library.

 

[FredBerlin]

Exactly. Should be no problem - 100% agreed.

Still, I must be missing a detail, just don't see it.

 

[agraham]

I assume you have tried this in release mode and the problem does persist then?

 

[FredBerlin]

Yes, I tried release and debug mode, and all possible settings for hardware flow control. No difference.

 

[Erel]

Better to monitor the input with:

Private Sub RxD_DataAvailable (Buffer() As Byte)
   Log("length: " & Buffer.Length)
   TotalLength = TotalLength + Buffer.Length 'global variable
   Log("Total: " & TotalLength)
End Sub

Is the length correct?

 

[FredBerlin]

Dear Erel,
the problem persists even when removing all costy operations and only adding
two integers (TotalLength + Buffer.Length) the way you proposed. I added a Log("New_Try____")
when the request button is pressed. The partial buffer lengths add up correctly, just obviously
sometimes bytes are not received into Buffer() (and also not counted then, of course).
Here is the logged output:

Buffer size: 16384
New_Try________________________
length: 27
--> Total: 27
length: 76
--> Total: 103
New_Try________________________
length: 38
--> Total: 38
length: 65
--> Total: 103
New_Try________________________
length: 31
--> Total: 31
length: 72
--> Total: 103
New_Try________________________
length: 35
--> Total: 35
length: 68
--> Total: 103
New_Try________________________
length: 50
--> Total: 50
length: 55
--> Total: 105
New_Try________________________
length: 35
--> Total: 35
length: 68
--> Total: 103
New_Try________________________
length: 87
--> Total: 87
length: 16
--> Total: 103
New_Try________________________
length: 52
--> Total: 52
length: 53

The point being: Technically it must be possible, because the App "Serial USB Terminal" (PlayStore) never looses any byte.
The only thing changed is the app used, everything else (hardware-wise) remains identical.

Since it is a half-duplex connection (FTDI chip, which does the TXD/RXD switching internally) my guess is, that there
could be an unwanted send/receive switching in-between the message chunks which would explain the loss of two bytes.
Is there a method to get errors like "overrun error", "framing error" or the like, to see if there happened a receive error
somewhere?

Best,
Fred

 

[agraham]

As far as I recall all the USB serial chips that I have ever come across, including FTDI chips, are full-duplex internally and externally and can all support simultaneous operation in both directions.

 

[agraham]

If you can, although it may not be possible as you say you cannot adjust the sending baud rate, I would try setting the sending device to add two stop bits. If this is not possible, as you are only dealing with ASCII set your receiver to seven data bits. Either of these should obviate the possibility of a framing error occurring.

Depending on the FTDI chip in the device would also try my original UsbSerial library if it supports that chip as it uses different low level USB code to felUSBSerial and I know that it has worked at high baud rates for me in the past.

 

[emexes]

Also the dropped bytes are at different places each run, that rules out decoding problems.

Do the dropped bytes usually/always occur at the boundary of/between the chunked data?
Eg, where you logged:

New_Try________________________
length: 87
--> Total: 87
length: 16
--> Total: 103

are the missing two bytes:
- at the start of the 103 bytes (ie, first two bytes are dropped)
- at the end of the 103 bytes (ie, last two bytes are dropped)
- in between the 87 and 16 byte chunks?
- within the 87 or 16 byte chunks?

Also, is device always coming back with the *exact* *same* 105 bytes in response to the list request when you use the other serial terminal programs?

Without seeing the output, I'm clutching at straws here, but: about once a decade I get tricked by something unexpected like tabs being expanded to spaces, or spaces being compressed to tabs.

Certainly the original feel of the problem was that it was a bit format mismatch, and that sometimes a byte would cause a framing error and thus miss a byte or two. Now that I think about it, the fact that it misses either zero or two bytes, but seemingly not just one byte, hints that it could be a resynchronization issue.

 

[emexes]

the fact that it misses either zero or two bytes, but seemingly not just one byte, hints that it could be a resynchronization issue.

Or perhaps the serial device using XON/XOFF flow control (could happen irrespective of whether the serial-to-USB adapter is configured to use same).

 

[emexes]

Another slightly radical idea: can you loop back the serial port ie connect RXD to TXD of the serial pins of the serial-to-USB adapter?

Then send yourself packets of say 30-to-250 random bytes, make sure you receive them back shortly thereafter.

Or can you generate test packets using another computer (laptop, even, with either a real serial port, or another USB-to-serial adapter). Or a different serial device - like, if you have an old modem hanging about, you could still have a "conversation" with it with AT commands, without needing to actually plug it into a phone line and make a call.

 

[FredBerlin]

This goes to agraham and emexes:

Dear agraham.
Yes, full duplex on the USB-side, half duplex on the RS485. The sending device is out of my control, I have to live with it.
Of course I could try to hook up test equipment to produce RS485 data stream to track it down, but that's kind of the last ressort.
I startet off with your library, but the device is so new that it isn't supported. If you tell me how to trick your library into using the
FTDI chip, I'd be happy to try out your library a second time. The post regarding the chip is here:
https://www.b4x.com/android/forum/threads/usbserial-supported-devices.104260/#content

Dear emexes.
To check what has been received actually I'd need to process the data bytes (I did this in the beginning)
but the latest tests as recommended by Erel throw away the data and just count the length. Maybe the next
to look into is to write those byte into a queue and only later read them out and display them. This way
timing issues are unlikely but I can still analyze where the missing bytes are.
I also thought about hooking up a scope, but the problem doesn't seem to be on the hardware side,
since other apps (or a PC connected) show everything just fine.
Best idea I came up with to test next: Prepare a bunch auf independent byte-buffers and copy each received
junk into one of them, so later on I have size + content for analysis.

Best,
Fred