Android Question parsing text that contain invalid charcters

Addo · Mar 8, 2018

i am trying to receive bytes and convert it to string and parse it

the bytes that is string list that saved to a memorystream and send to b4a Asyncstream client socket

the code of new data looks like following

B4X:

Public Sub NewData (data() As Byte)
 
Dim msg As String

msg = BytesToString(data, 0, data.Length, "UTF8")

Dim param As String = msg
Dim paramnum() As String = Regex.Split("\~", param)

Log(paramnum(0))

End Sub

the text that sent from server looks like this

url1~
url2~
Url3~

and so on but after received and convert bytes to string there is some invalid charcters inserted that break the parsing like following

B4X:

url1~
������������
url2~
Url3~

i dont know from where this characters came after bytetostring conversion

now when i try to capture the parsing data as example paramnum(2)
i got an exception

main_vvvvvvvvv4 (java line: 489)
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2
at app.name.main._vvvvvvvvv4(main.java:489)
at app.name.main._astream_newdata(main.java:381)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:186)
at anywheresoftware.b4a.BA$2.run(BA.java:360)
at android.os.Handler.handleCallback(Handler.java:743)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:150)
at android.app.ActivityThread.main(ActivityThread.java:5621)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:684)
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2

how can i solve this invalid charter after conversion ?

i have used msg = msg.trim with no luck to solve

also i have try to capture if there is any hidden charter that could break that so i used check hidden charcters online i come out with this result

B4X:

URL1~

&#65279;URL2~
URL3~
URL4~
URL5~

MarkusR · Mar 8, 2018

the example used "UTF-8" i don't know if this will be a difference. you used "UTF8" above

Addo · Mar 8, 2018

I am already used utf8 in both sides

MarkusR · Mar 8, 2018

i meant
i just saw in b4a was it mentioned with a minus char. your code above is without.

Addo · Mar 8, 2018

I don't get your point a code explaination would help where in my code shows minus char ?

Addo · Mar 8, 2018

If you are talking about ~ this is not a minus and its already in the split function .

DonManfred · Mar 8, 2018

No, he is talking about that you use "UTF8" and usually it is named "UTF-8"

PassionDEV said:

It would be helpful if you post the data received as HEX. If it contains the Bytes 0xef,0xbb,0xbf then it is probably a BOM-Header.

Can you upload a textfile with the data you are transfering?

Addo · Mar 8, 2018

Hex data attached as image for better view

DonManfred · Mar 8, 2018

http://www.fileformat.info/info/unicode/char/feff/index.htm

Looks like it is a BOM.... Make sure the text you are sending is UTF8 but NOT "UTF8 with BOM".

Again: Please Upload a TEXT FILE (NOT JPG) of the text you are sending from the sending side.

DonManfred · Mar 8, 2018

I did not asked for a code you are using.
Write the data to a textfile instead of a stream you are sending. Or write the data to a stream which points to a file.
Upload this file.

All i can say is that the data looks like to have a Byte order Mark (BOM) which it should NOT.

DonManfred · Mar 8, 2018

You can try to use the BytesBuilder and remove an BOM from the Stream (byte array in new_data) before converting the bytes to a string.

Addo · Mar 8, 2018

here is the saved data from server side after sent

Addo · Mar 8, 2018

DonManfred said:
You can try to use the BytesBuilder and remove an BOM from the Stream (byte array in new_data) before converting the bytes to a string.

this will corrupt the bytes data and the results will not be showing as expected

klaus · Mar 8, 2018

This is the hex content of your file:

The first three characters are, as DonManfred suggested, the BOM characters.
Then you have 2SCLIENT~.
And a CR and a LF character after each ~ character.

Addo · Mar 8, 2018

i have tried with no luck

B4X:

msg = BytesToString(data, 0, data.Length, "UTF-8")
msg = msg.Trim
msg = msg.Replace(CRLF, "")
msg = msg.Replace(Chr(10), "")
msg = msg.Replace(Chr(13), "")
msg = msg.Replace(Chr(127), "")

Erel · Mar 8, 2018

Why are there invalid characters in the data? If the data doesn't represent a string then you shouldn't convert it to a string.

Use BytesBuilder instead.

klaus · Mar 8, 2018

The code below works with you file:

B4X:

Private txt, split() As String

txt = File.ReadString(File.DirAssets, "mss.txt")
Log (txt)
split = Regex.Split("~" & Chr(13) & Chr(10), txt)
For i = 0 To split.Length - 1
    Log(split(i))
Next

In the file you sent, there are no other 'invalid' characters like in the text you show in your firt post.

Addo · Mar 8, 2018

because it was pure text from server side does not process as byetstostring
iam still trying to figure how bytesbuilder works in this case

klaus said:
The code below works with you file:

B4X:

Private txt, split() As String txt = File.ReadString(File.DirAssets, "mss.txt") Log (txt) split = Regex.Split("~" & Chr(13) & Chr(10), txt) For i = 0 To split.Length - 1 Log(split(i)) Next

In the file you sent, there are no other 'invalid' characters like in the text you show in your firt post.

Addo · Mar 8, 2018

@Erel i am doing following

B4X:

bb.Append(data)
bb.Remove()

but remove part what i have to remove ? U+FEFF ? i still trying to figure out

klaus · Mar 8, 2018

Well, I am a bit surprised!
Why do you sent something which doesn't show your problem!

Android Question parsing text that contain invalid charcters

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Well-Known Member

Expert

Well-Known Member

Attachments

Expert

Expert

Expert

Well-Known Member

Attachments

Well-Known Member

Expert

Well-Known Member

B4X founder

Expert

Well-Known Member

Well-Known Member

Expert

Similar Threads

Privacy & Transparency

Privacy & Transparency