Android Question [felUSBSerial] Missing data while receiving bytes

FredBerlin

Member
Licensed User
Hello everybody!
After using felUSBSerial for some while, I realized that once a while some bytes are lost
in the input stream. The procedure is as follows:
Clicking the button "SEND" a short message is transmitted over an FTDI-cable to an external
device. The device then answers with a byte message (simple ASCII chars) of a few hundred bytes.
While receiving the continuous input stream (all bytes are chained together, i.e., after each stop bit the
next start bit starts immediately) always some bytes are lost a random positions.
Here is the code:

B4X:
Sub Process_Globals
    Private usbserial As felUsbSerial
    Private manager As UsbManager
    Private bc As ByteConverter
End Sub

Sub Globals
    Private txtMessage As EditText
    Dim btnConnect As Button
    Dim btnClose As Button
    Dim btnSend As Button
End Sub

Sub Activity_Create(FirstTime As Boolean)
    Activity.LoadLayout("Test_felUsbSerial")
    If FirstTime Then
        manager.Initialize
    End If

    ' Dim btnConnect As Button
    btnConnect.Initialize("btnConnect")
    btnConnect.Text = "Connect"
    Activity.AddView(btnConnect, 10dip, 10dip, 100dip, 40dip)

    ' Dim btnClose As Button
    btnClose.Initialize("btnClose")
    btnClose.Text = "Close"
    Activity.AddView(btnClose, 120dip, 10dip, 100dip, 40dip)

    ' Dim btnSend As Button
    btnSend.Initialize("btnSend")
    btnSend.Text = "Send"
    btnSend.Enabled = False
    Activity.AddView(btnSend, 230dip, 10dip, 100dip, 40dip)
End Sub

Sub btnConnect_Click
    If manager.GetDevices.Length = 0 Then
        Log("No connected usb devices.")
        btnSend.Enabled = False
    Else
        Dim device As UsbDevice = manager.GetDevices(0) 'assuming that there is exactly one device
        If manager.HasPermission(device) = False Then
            btnSend.Enabled = False
            ToastMessageShow("Please allow connection and click again.", True)
            manager.RequestPermission(device)
        Else
            usbserial.BUFFER_READ_SIZE = 16 * 1024 ' different settings make no difference
            usbserial.Initialize("RxD", device, -1)
            usbserial.BaudRate = 57600
            usbserial.DataBits = usbserial.DATA_BITS_8
            usbserial.StopBits = usbserial.STOP_BITS_1
            usbserial.Parity = usbserial.PARITY_NONE
            usbserial.FlowControl = usbserial.FLOW_CONTROL_OFF ' tested all settings here
            usbserial.StartReading
            btnSend.Enabled = True
        End If
    End If
End Sub

Sub btnClose_Click
    If usbserial.IsInitialized Then
        usbserial.Close
        btnSend.Enabled = False
    End If
End Sub

Sub btnSend_Click
    If usbserial.IsInitialized Then
        Dim s As String = "list" & Chr(13)
        Dim msg() As Byte = s.GetBytes("UTF8")
        txtMessage.Text = ""
        usbserial.Write(msg)
    End If
End Sub

Private Sub RxD_DataAvailable (Buffer() As Byte)
    txtMessage.Text = txtMessage.Text & bc.StringFromBytes(Buffer, "UTF8")
    ' even if we don't add the new chars, but only show the last received ones, still there are lost chars,
    ' so seems like processing time of the txtMessage rendering doesn't seem to be the problem either
End Sub

Now the whole communication works properly using a standard terminal program on the same mobile
device, i.e., many things can be ruled out: the missing bytes are not due to baud rate mismatch,
half duplex switching, driver problems or the like.

Any ideas?
Best,
Fred
 

emexes

Expert
Licensed User
I'd hook that 'scope up. At worst, it'll give you something to look at while you mull over what could be causing bytes to go AWOL. At best, it might trigger a random thought that leads to a solution.

RS485 brought back memories about timing issues with the transmit enable - many UARTs generate a transmit (buffer empty) interrupt after the last data bit has been sent, but before/whilst the stop bit is sent, which *shouldn't* be a problem but... if the bus isn't terminated, then perhaps there is a glitch that gets misinterpreted as a start bit, something like that.

It is good that you have software that is apparently not dropping bytes, because this means that what you want to do is indeed actually possible; all we need to do is find out the reason for the difference. How hard can that be?!?! Perhaps the serial terminal programs are more relaxed about framing errors, whereas the felUSBSerial is rejecting such marginal cases and dropping the bytes.
 
Upvote 0

FredBerlin

Member
Licensed User
Hi guys,

further testing gave more insights:
1. the missing bytes always come as two bytes in series, in the middle of the Byte() array,
that the message is split into junks has nothing to do with it
2. the RS485 is properly terminated, the signal on the scope is all good
3. if I switch to an older interface (ftdi-usb-chip): no missing bytes at all !!!

To sum up:
1. problems are gone if I switch to an older interface (not an option, since the old one is no longer produced)
2. problems are gone if I switch to Android Studio (which I hate to do, rather would like to stay in B4A)
3. problems may be gone switching from felUSBSerial to USBSerial, but the newer chip isn't supported in USBSerial

At the time being I wonder if I either look into "writing a library for B4A", to move the working code from Android Studio
to B4A, or if that may take longer than coding the whole app in Java (which doesn't attract me that much).

Any suggestions? Further analysis requested? Your opinion on how to proceed further?

Thank you all so far,
best,
Fred
 
Upvote 0

agraham

Expert
Licensed User
Longtime User
So your rather thorough analysis seems to point to a bug in the lower level driver part of felUSBSerial when used with a newer FTDI chip :(

If it were me I would look to encapsulate the known working USB side of your Android Studio code as a B4A library. As you already have working code in Android Studio it really should be quite a simple task, much easier than moving the whole app to Java, and as a bonus you would know how to write B4A libraries should you need to in future :)

I don't know much about Android Studio as I used to write my libraries using Eclipse before I gave up using Android a few years ago. So I don't know if Android Studio can produce jar files for use as B4A libraries. Having recently resumed using B4A I have updated and recompiled a couple of my existing libraries and created quite a complex new one using only Notepad++ to edit the code and Erel's Simple Library Compiler to compile it.
 
Upvote 0

FredBerlin

Member
Licensed User
That sounds like a plan. Always hesitating to install another huge package (Eclipse), I'm -like you- in favour of small and neat solutions.
Notepad++, JAVA, and Erel's "Simple Library Compiler" may be appealing. I'll find the first two on my own. And Erel's compiler by searching
the forum I guess.
 
Upvote 0

FredBerlin

Member
Licensed User
Meanwhile I used a PC and RealTerm to send a text file to felUSBSerial (57600 baud, 8N1).
Here my findings: If I include 500µs idle time between the chars (one can easily do this with RealTerm)
there are no errors at all with felUSBSerial and the new FTDI chip, but removing the idle time and
sending it as a continuous stream, the drop outs (always as pair of two bytes) are back in.
So really appears to be a low-level driver problem. This information may help others in the (near) future.
Those new FTDI chips may definitely be used by a growing number of product suppliers/users.
 
Upvote 0

agraham

Expert
Licensed User
Longtime User
As an example of how easy it it to make a library here is a zip of a folder that will compile my Jpeg library. Just unzip it anywhere, navigate to the Jpeg folder in File Explorer and double click CompileJpeg.bat. You will end up with a shiny new library in your Additional Libraries folder.
 

Attachments

  • Libraries.zip
    174.2 KB · Views: 228
Upvote 0

FredBerlin

Member
Licensed User
Cool! Thank you so much. That will definitely shorten my development time.
In the end the serial interface will be just a tiny bit of a larger app that will process the data.
The sooner I get to the main project, the better.
 
Upvote 0

FredBerlin

Member
Licensed User
There is highly likely something going wrong in felUSBSerial with the new FTDI chip.
While doing further diagnoses to find out when missing bytes start to occur (I inserted less/more idle time between bytes sent) I once a while got this log msg:
New_Try________________________
length: 68
--> Total: 68
length: 75
--> Total: 143
length: 74
--> Total: 217
New_Try________________________
length: 36
--> Total: 36
length: 56
--> Total: 92
length: 56
--> Total: 148
length: 62
--> Total: 210
length: 13
--> Total: 223
New_Try________________________
length: 15
--> Total: 15
java.lang.ArrayIndexOutOfBoundsException: length=61; index=61
at com.felhr.usbserial.FTDISerialDevice$FTDIUtilities.copyData(FTDISerialDevice.java:594)
at com.felhr.usbserial.FTDISerialDevice$FTDIUtilities.adaptArray(FTDISerialDevice.java:502)
at com.felhr.usbserial.UsbSerialDevice$WorkerThread.run(UsbSerialDevice.java:248)
length: 54
--> Total: 69
If I read this correctly, it stems from deep within felUSBSerial. Index problems could also explain missing bytes! So it seems that there's a strange interplay between the new FTDI chip and the current version of felUSBSerial - definitely this chip has to be on the "is not yet supported" list of SerialUSB AND felUSBSerial at the time being. :-(
 
Upvote 0

FredBerlin

Member
Licensed User
Now here is the idea: comparing the old FTDI chip and the new one, it pops into the eye that the max. pack. size is 512 instead of 64.
Let's say the chip throws an interrupt a few bytes before the buffer is full (default threshold may be 4), then the old chip delivers chunks
of around 60 bytes max, whereas the new one could in principle go up to 508 bytes.
The crash occurs always when Array-index is 61... go figure.

_________ WORKING ______NOT WORKING
Manufact.:_FTD____________FTDI
Product:___FT2____________USB <-> Serial Converter
Serial:_____AI0____________FT3FSMK2
DeviceID:__0x3EA__________0x3EA
Prod.ID:___0x6001_________0x6014
VendorID:__0x403__________0x403
max. Pack.:_64_____________512

Since this information has been read by the app "USB Device Info" (to be found on F-Droid), the chip directly reports
the needed buffer size (and did this already in the past). So my guess is, that felUSBSerial could be even more versatile
and future-save, if this property is actually read and used for buffer allocation.

Erel, just in case you are in the mood to set the buffer size to 512 within felUSBSerial,
I'd be happy to re-run all tests and report on it.

Meanwhile I look into library writing which may be helpful anyway ;-)

Best,
Fred
 
Upvote 0
Top