iOS Question StringsUtil UTF-8 Encoding/Decoding

[iCAB]

Hi All

I am using the following code to encode/decode messages sent/received by the app.

The code works perfectly ok with B4A, but not with B4I
The issue is more apparent when a message sent from B4A to B4I.

su.EncodeUrl(Parameters(i), "UTF8")

below is the code I am using to decode

#if B4A
    TLocalPS.Command = su.DecodeUrl(Message.GetData.Get("body"), "UTF8")
#else   
    TLocalPS.Command = su.DecodeUrl(Message.Get("body"), "UTF8")
#end if

with B4I I am left with unwanted characters, so I find myself having to call this after the executing the above code.

CleanedMessage = CleanedMessage.Replace("%40", "@").Replace("%2C", ",").Replace("+", " ").Replace("%2F", "/").Replace("%3A", ":").Replace("%20", " ")

is there a cleaner way of doing this?

Thanks
iCAB

 

[Erel]

How does the input look like?

Why do you need to url encode the message? How is it sent?

 

[iCAB]

Hi Erel

Thanks for your reply. Ok let me clarify a bit more

The app has a communication wrapper and multiple ways of sending a message.

• http to a web service
• websocket to b4J server
• Firebase messaging

all messages formatted using http like format:

http://someservice?Parm1=xxx&Parm2=yyy&Parm3 =xxx etc...

The wrapper encodes the message parameters using UTF-8 (by default ) as the destination is programmable by the server on the fly.

Here is the code used to encode the message

Public Sub JobCreateParmList( Parameters() As String ) As String
    Dim su As StringUtils
    Dim sb As StringBuilder
    Dim strRequest As String
    
    sb.Initialize
    
    For i = 0 To Parameters.Length - 1 Step 2
        If i > 0 Then sb.Append("&")
        If Parameters(i).Contains("=") = False And Parameters(i).Contains(">") = False And Parameters(i).Contains("<") = False Then
            sb.Append(su.EncodeUrl(Parameters(i), "UTF8")).Append("=")
        Else
            sb.Append(su.EncodeUrl(Parameters(i), "UTF8"))
        End If
        sb.Append(su.EncodeUrl(Parameters(i + 1), "UTF8"))
    Next
    
    strRequest = sb.ToString

    Return strRequest
End Sub

When the message is received (say using web socket or FCM), I am calling

su.DecodeUrl(Message.Get("body"), "UTF8")

The above code, leaves the unwanted characters in the message

Thanks
iCAB

 

[Erel]

You can use Job.Download2 to encode the parameters.

Can you post an example of the input that isn't decoded properly?

 

[iCAB]

Hi Erel

Sorry for the late reply, but I thought it will be easier with a sample project that shows the end to end operation

How to use:

• Look for SendTestFCMMessageToiOS
• Input your FCMKey & iOS Topic

I also included this function to show you how I am decoding the message

• FCM_MessageArrived

To see the issue, use the attached project to send a message to an iOS device

Important to note that I am calling: JobCreateParmList, instead of Download2. The code copied from Download2 and used to encode the parameters only



Thank you

 

[iCAB]

You can use Job.Download2 to encode the parameters.
Can you post an example of the input that isn't decoded properly?

Hi Erel

Did you get a chance to check the above?

Thanks
iCAB

 

[Erel]

It will be easier if you post the input string.

 

[iCAB]

It will be easier if you post the input string.

Hi Erel

Here is the function used in the attached test code

Private Sub SendTestFCMMessageToiOS( )
    
    Dim Parameters() As String =  Array As String ( "func", "hello", _
                                                    "Email", "[email protected]", _
                                                    "Msg", "This is a test") ', _
    
    Dim Job1 As HttpJob
    Job1.Initialize("Ignore", Me)
    
    Dim MsgBody As String = Job1.JobCreateParmList( Parameters )
    Dim FCMKey As String = "AIza....."
    Dim Topic As String = "iOS_Topic."
    
    HTTPW_SendFCMRequest( Topic, MsgBody, FCMKey)
    
End Sub

As a result: MsgBody =

func=hello&Email=someemailaddress%40somedomain.com&Msg=This+is+a+test

When decoded with B4A, the data looks exactly as in the Parameters array. When decoded with B4I: %40 & "+" stay in ( Sure, I can use replace later on, but I was hoping for a cleaner approach)

Hopefully this clarifies what I am looking for

Thanks
iCAB

 

[Erel]

This is what I get from this code:

Dim s As String = "func=hello&Email=someemailaddress%40somedomain.com&Msg=This+is+a+test"
Dim su As StringUtils
Log(su.DecodeUrl(s, "utf8"))

func=hello&[email protected]&Msg=This+is+a+test

There are several types of url encoding and the + sign is not always an escaped string.

If you change the url to:

Dim s As String = "func=hello&Email=someemailaddress%40somedomain.com&Msg=This%20is%20a%20test"

Then it will work as expected.

The bottom line is that you should manually replace the + characters with spaces.

 

[iCAB]

T
The bottom line is that you should manually replace the + characters with spaces.

Thanks Erel,
That's exactly what I am doing (on the receiving end). The only issue, that it is not only " ", many other characters such "@", etc..

Not a big deal, It's working

Thanks again

 

[OliverA]

The bottom line is that you should manually replace the + characters with spaces.

It still would be nice if StringUtils EncodeUrl/DecodeUrl would work consistently across all three platforms. Since it does not, one now as to keep that in mind when performing encoding in B4J and B4A, by running an additional step (replacing +'s with %20's) after using EncodeUrl.

To me, the code below should work the same on all three platforms (but does not):

Dim su As StringUtils
Log(su.EncodeUrl("This is a test", "UTF8"))
Log(su.DecodeUrl(su.EncodeUrl("This is a test", "UTF8"), "UTF8"))
Log(su.DecodeUrl("This%20is%20a%20test", "UTF8"))
Log(su.DecodeUrl("This+is+a+test", "UTF8"))

B4A output:

This+is+a+test
This is a test
This is a test
This is a test

B4J output:

This+is+a+test
This is a test
This is a test
This is a test

B4i output:

This%20is%20a%20test
This is a test
This is a test
This+is+a+test

One could actually ignore iStringUtil's different encoding (using %20 instead of +) if it at minimum it would properly decode the +'s.

 

[Erel]

I will fix it. Did you encounter any other character that is not decoded in B4i (ignore the encoding as the output is valid)?

 

[OliverA]

Tell you the truth, I've not tried any other characters. Doing some searches on this issue, it looks like the + is the sticking point (issue) when it comes to iOS handling encoding/decoding of URL information.

The B4J/B4A StringUtils' EncodeUrl/DecodeUrl are designed for "application/x-www-form-urlencoded" data. B4i's methods are more EncodeParametersOfAURL/DecodeParametersOfAURL functions then total URL encode/decode functions, since certain symbols are allowed before a ? of a URL then after the ? of a URL. Amazing how complicated something can get.

Sources:
https://stackoverflow.com/a/1211256
https://stackoverflow.com/a/7920424
https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4.1
https://www.ietf.org/rfc/rfc2396.txt

 

[OliverA]

Look for SendTestFCMMessageToiOS

From what I can tell, you use JobCreateParamList to create a parameter list. From the code flow I can see, it does not need to be URL encoded (you're just decoding it in FCM_MessageArrived), so you could decode it before calling HTTPW_Send_FCMRequest(Topic, MsgBody, FCMKey) (and do away with the decoding in FCM_MessageArrived).

   Dim MsgBody As String = Job1.JobCreateParmList( Parameters )
   Dim FCMKey As String = "AIza....."
   Dim Topic As String = "iOS_Topic."
   Dim su As StringUtils
   MsgBody = su.DecodeUrl(MsgBody, "UTF8")
   HTTPW_SendFCMRequest( Topic, MsgBody, FCMKey)

I'm not saying you did not find an issue (you did), but this way the decoding happens on the same platform as the encoding.

 

[iCAB]

Hi Oliver

I am fully aware of what you are saying, and I already mentioned that.
If you look closely at the third post you can see what I am trying to achieve.
The application communication protocol is fully programmable by the server, and how messages are sent.
The same message can be sent via:

• http
• WebSocket
• or directly using FCM

I know that I can put the necessary hooks to customize this, however I was trying to avoid that.


Thanks for your input.
Greatly appreciated

 

[iCAB]

I will fix it. Did you encounter any other character that is not decoded in B4i (ignore the encoding as the output is valid)?

Hi Erel

So far this is what I am using:

CleanedMessage = CleanedMessage.Replace("%40", "@").Replace("%2C", ",").Replace("+", " ").Replace("%2F", "/").Replace("%3A", ":").Replace("%20", " ")

 

[Erel]

So far this is what I am using:

It is not needed. You can see it with this code:

Dim s As String = "%40 %2C + %2F %3A %20"
Dim su As StringUtils
Log(su.DecodeUrl(s, "utf8"))

The output is:

@ , + / :

Only the + sign is not converted back to space.

 

[iCAB]

It is not needed. You can see it with this code:.

Hi Erel

It happens when you are sending from B4A to B4I.
If you use the project I supplied and use the code below you will see the issue

    Dim Parameters() As String =  Array As String ( "func", "hello", _
                                                    "Email", "[email protected]", _
                                                    "Msg", "@, / :" ) ', _

Here is a screenshot from the receiving end

Thanks
iCAB

 

[OliverA]

What does the string look like before you decode it and after (in your iOS applicatoin)?

 

[OliverA]

#if B4A
    TLocalPS.Command = su.DecodeUrl(Message.GetData.Get("body"), "UTF8")
#else   
    log(Message.Get("body")) ' <----
    log(su.DecodeUrl(Message.Get("body"), "UTF8")) '<----
    TLocalPS.Command = su.DecodeUrl(Message.Get("body"), "UTF8")
#end if