iOS Question MQTT adds extra characters when encrypted

[aaronk]

Hi,

Not sure where to post this (B4J or B4i forum) so I posted it here, as I am not sure what end is causing my issue.

I am using MQTT to send data from my B4J app (non-UI) to my B4i app.

Sending the text to the topic in plain text and receiving it in B4i works fine.

Soon as I encrypt the message it is causing an issue.

I am using the following code..

B4J:

Type MQTTMSG (message As String)

' sending MQTT message
MqttClient.Publish2("test", CreateMessage(msg), 0,False)

Public Sub EncryptText(text As String) As String
    Dim c As B4XCipher
    Return su.EncodeBase64(c.Encrypt(text.GetBytes("utf8"), "SecurePassword"))
End Sub

public Sub CreateMessage(msg As String) As Byte()
    Dim m As MQTTMSG
    m.Initialize
    m.message = EncryptText(msg)
    Return serializator.ConvertObjectToBytes(m)
End Sub

B4i:

Type MQTTMSG (message As String)

Public Sub DecryptText(EncryptedData As String) As String
    Dim c As Cipher 'iEncryption Lib
    Private su As StringUtils
    Dim b() As Byte = c.Decrypt(su.DecodeBase64(EncryptedData), "SecurePassword")
    Return BytesToString(b, 0, b.Length, "utf8")
End Sub

Private Sub MqttClient_MessageArrived (Topic As String, Payload() As Byte)
    Log("MQTT MessageArrived")
    Dim receivedObject As Object = serializator.ConvertBytesToObject(Payload)
    Dim m As MQTTMSG = receivedObject   
    Dim msg1 As String = DecryptText(m.message)
    Log("msg length = " & msg1.Length) ' logs 48
    Log(msg1) ' logs, as per image below
End Sub

For some reason, it is adding extra characters to the end of the message from what I can see.

Like I said, when I send it as plain text without encrypting the message it works fine.

Any ideas on what I have done wrong ?

 

[Erel]

What is the output of Log(msg) in B4J?

What is the actual text supposed to be?

 

[emexes]

For some reason, it is adding extra characters to the end of the message from what I can see.

Base64 encoding is done in groups of three characters. Shouldn't be a problem because it is supposed to be padded out (with "="s) but perhaps there is some variation between the B4i and B4J inplementations.

One test would be to, on the B4J transmitting side, to do a complete decode/decrypt/deserialize/etc of the Byte Array output of CreateMessage(), compare it to the original.

Also compare the lengths of the corresponding interim stages on each side (B4J & B4I) ie, which step of the process is adding the extra characters?

 

[aaronk]

What is the output of Log(msg) in B4J?

It's the same as what is sent without the extra invisible cha

What is the actual text supposed to be?

3670BB9F//16RR22200732409191110069

I added to the log:
Log("msg length = " & msg.Length)
So I can see the length of the message as well as what is sent.

B4J (which is sending the message):

B4i (Getting the message):

After looking into this in more detail, it looks like the msg length is not matching what is being sent. I think it might be adding CRLF or something to the end of the message, when using the encryption.

However if I bypass (remove the encryption) it works fine with the exact same message.

On the B4i end (when not sending the message with the encryption), the message it shows is the same, but B4i shows the length is only 36.

Seems to be something to do with the encryption for some reason.

 

[emexes]

It is curious that the ratio of the msg lengths 48/36 is exactly the same as the expansion factor of Base64 encoding 4/3.

Are the message string texts and lengths logged consecutively, and of the same variable?

 

[aaronk]

Are the message string texts and lengths logged consecutively, and of the same variable?

Yes, the same message and sent one after the other.

 

[emexes]

Well, that is mighty interesting. Could you log the interim results, eg using the code below, taken from post #1 but with the transformations done and logged one-by-one, and using your variable names where already present:

B4J:

Type MQTTMSG (message As String)

' sending MQTT message
MqttClient.Publish2("test", CreateMessage(msg), 0,False)

Public Sub EncryptText(text As String) As String
    Dim c As B4XCipher

    Log("text " & text.Length & " = [" & text & "]")

    Dim B1() As Byte = text.GetBytes("utf8")
    Log("B1 " & B1.Length)

    Dim B2() As Byte = c.Encrypt(B1, "SecurePassword")
    Log("B2 " & B2.Length)

    Dim S3 As String = su.EncodeBase64(B2)
    Log("S3 " & S3.Length & " = [" & S3 & "]")

    Return S3
End Sub

public Sub CreateMessage(msg As String) As Byte()
    Dim m As MQTTMSG
    m.Initialize
    m.message = EncryptText(msg)

    Dim B4() As Byte = serializator.ConvertObjectToBytes(m)
    Log("B4 " & B4.Length)

    Return B4
End Sub

B4I:

Type MQTTMSG (message As String)

Public Sub DecryptText(EncryptedData As String) As String
    Dim c As Cipher 'iEncryption Lib
    Private su As StringUtils

    Log("EncryptedData " & EncryptedData.Length & " = [" & EncryptedData & "]")

    Dim B2() As Byte = su.DecodeBase64(EncryptedData)
    Log("B2 " & B2.Length)

    Dim b() As Byte = c.Decrypt(B2, "SecurePassword")
    Log("b " & b.Length)

    Dim S1 As String = BytesToString(b, 0, b.Length, "utf8")
    Log("S1 " & S1.Length & " = [" & S1 & "]")

    Return S1
End Sub

Private Sub MqttClient_MessageArrived (Topic As String, Payload() As Byte)
    Log("MQTT MessageArrived")
    Dim receivedObject As Object = serializator.ConvertBytesToObject(Payload)
    Dim m As MQTTMSG = receivedObject

    Dim msg1 As String = DecryptText(m.message)
 '''Log("msg1 " & msg1.Length = [" & msg1 & "]")        'wrong :-/ nice try, no cigar
    Log("msg1 " & msg1.Length & " = [" & msg1 & "]")    'corrected ;-)
  
End Sub

 

[aaronk]

B4J Logs:

B4i Logs:

Looks like the encrypted data is the same as the receiving side.

 

[emexes]

What it looks like is that:

B4J c.Encrypt converts 36 bytes (B1) to 72 bytes (B2)

B4I c.Decrypt unconverts that 72 bytes (B2) back to 48 bytes (b)

Well, that's interesting ;-)

I'll try it here, see if I get the same thing.

 

[emexes]

edit: never mind, problem/solution found: https://www.b4x.com/android/forum/threads/share-encrypted-data-with-b4a.35482/

I'll try it here, see if I get the same thing.

Turns out I've got my own set of problems ;-)

I probably have mismatching B4J and JDK versions or something like that.

 

[emexes]

On the bright side, though, it did lead me a little bit down the "how does this encryption thing work anyway" trail, and I remembered that it probably does encryption in blocks, and thus the data being encoded would have to be padded out to a multiple of the block size. Which would make your results make sense, if the block size was 24 bytes: your 36 bytes would be padded out to 2 full blocks ie 48 bytes, and then upon decryption you get back 48 bytes which is your original 36 bytes plus 12 bytes of padding.

But I would have expected this discrepancy to be clearly documented in the encrypt/decrypt function documentation, because it is something that every user would encounter. Hmm...

 

[emexes]

I did the entire chain of string -> bytes -> encrypt -> base64 and back again in B4J, and got back precisely the original string ie 36 characters (34 printable, plus CR LF).

So now to check same in B4I. But I was trying to add another test device yesterday, got bogged down with certificates/keys/etc, so I am expecting this to go about as well as a visit to the dentist :-/

 

[emexes]

Righto, the problem is that the B4I Cipher.Decrypt routine is not removing trailing padding of Chr(0).

B4J B4XEncrypt.Decrypt does not add nulls when decrypting encryptions done by both B4J and B4I
B4I Cipher.Decrypt adds nulls when decrypting encryptions done by both B4J and B4I

Workaround is:

Dim FirstPadding As Int = PlainText.IndexOf(Chr(0))
If FirstPadding > 0 Then
    PlainText = PlainText.SubString2(0, FirstPadding)
End If

Test code was:

Sub TestDecrypt

    Dim c As Cipher
    Dim su As StringUtils
 
    'B4J generated:
    'S3 = "tf1tTojxcBq1lGjrEagmRUcgD3QqLWgfQUUKGTsQVGjIwd+CCjR7OWljey6eArSJ5ch2iJVZe8/u4zi/wQ+qx6nvB6JWix8q"
    'S3 = "qLouEbyuNrmPbdYD0EtKDSSAO2HEZEs+glMKPJscxrGLXfArsIBF9NCzNF8fADaVr7fK6Hz+bmRXRRTEKP0XZUgiqAJcJ1Fd"
    'S3 = "QyvnUINMXQ17SXHuVx6oAYE4TOmxVHc+vRkDROlwbBD/ycHJB21EwzEGRwy/cbW/K7755Nn63g7SZv4W0NKqDiqyIWJvTwjj"

    'B4I generated:
    'S3 = "AB6Fcx31/JK5FCu1uUnGjJxhyLkoC5C82sfL92k+nUMcxOfqf8n83rsLQ7LFWo4tT0P6LjujCvfrylaN22GYgqlUFfYHnCUy"
    'S3 = "Bw+pdQoWFV4LrreX4B2eQA7LjbmrOS9dKGJ/e7jhUIRLBQSRBgmhgOCmgckLA6z2O3nbS/UAnWl5pVQqYq7Gad+m0Q1rn+om"
    'S3 = "zgJ3/3UzB8ea6E7fLn68meNl0JSACPZGLQgyE0H5RC9/5ts+xgh7kA8au3C57bQtd1xMRtEXj3SBCTUFid6WgW+i8fzCwEMx"

    Dim TestInput() As String = Array As String( _
        "tf1tTojxcBq1lGjrEagmRUcgD3QqLWgfQUUKGTsQVGjIwd+CCjR7OWljey6eArSJ5ch2iJVZe8/u4zi/wQ+qx6nvB6JWix8q", _
        "qLouEbyuNrmPbdYD0EtKDSSAO2HEZEs+glMKPJscxrGLXfArsIBF9NCzNF8fADaVr7fK6Hz+bmRXRRTEKP0XZUgiqAJcJ1Fd", _
        "QyvnUINMXQ17SXHuVx6oAYE4TOmxVHc+vRkDROlwbBD/ycHJB21EwzEGRwy/cbW/K7755Nn63g7SZv4W0NKqDiqyIWJvTwjj", _
        "AB6Fcx31/JK5FCu1uUnGjJxhyLkoC5C82sfL92k+nUMcxOfqf8n83rsLQ7LFWo4tT0P6LjujCvfrylaN22GYgqlUFfYHnCUy", _
        "Bw+pdQoWFV4LrreX4B2eQA7LjbmrOS9dKGJ/e7jhUIRLBQSRBgmhgOCmgckLA6z2O3nbS/UAnWl5pVQqYq7Gad+m0Q1rn+om", _
        "zgJ3/3UzB8ea6E7fLn68meNl0JSACPZGLQgyE0H5RC9/5ts+xgh7kA8au3C57bQtd1xMRtEXj3SBCTUFid6WgW+i8fzCwEMx" _
    )
 
    For I = 0 To TestInput.Length - 1
        Dim S3 As String = TestInput(I)

        Log("S3 " & S3.Length & " = [" & S3 & "]")
     
        Dim XB2() As Byte = su.DecodeBase64(S3)
        Log("XB2 " & XB2.Length)
     
        Dim XB1() As Byte = c.Decrypt(XB2, "SecurePassword")
        Log("XB1 " & XB1.Length)
     
        Dim NumberString As String = XB1(0)
        For J = 1 To XB1.Length - 1
            NumberString = NumberString & " " & XB1(J)
        Next
        Log(NumberString)
     
        Dim Xtext As String = BytesToString(XB1, 0, XB1.Length, "utf8")
        Log("Xtext " & Xtext.Length & " = [" & Xtext & "]")
    Next

End Sub

produces log:

Application_Start
S3 96 = [tf1tTojxcBq1lGjrEagmRUcgD3QqLWgfQUUKGTsQVGjIwd+CCjR7OWljey6eArSJ5ch2iJVZe8/u4zi/wQ+qx6nvB6JWix8q]
XB2 72
XB1 48
51 54 55 48 66 66 57 70 47 47 49 54 82 82 50 50 50 48 48 55 51 50 52 48 57 49 57 49 49 49 48 48 54 57 13 10 0 0 0 0 0 0 0 0 0 0 0 0
Xtext 48 = [3670BB9F//16RR22200732409191110069

Paste to forum editor stops at first Chr(0) but there's enough there to prosecute the case given that B4J B4XCipher.Decrypt is not padding with trailing Chr(0), and issue occurs (or not) regardless of whether encryption done by B4J B4XCipher.Encrypt or B4I Cipher.Encrypt.

 

[emexes]

So now to check same in B4I. But I was trying to add another test device yesterday, got bogged down with certificates/keys/etc, so I am expecting this to go about as well as a visit to the dentist :-/

This little flurry occurred whilst I was offline just now, brought back memories, almost made me smile :-/

 

[aaronk]

Ended up using the following code in B4i and it seem to fix the issue.

msg1 = msg1.Replace(Chr(0),"")

 

[emexes]

msg1 = msg1.Replace(Chr(0), "")

Nice! My view of the forest was blocked by trees :-/

 

[Sandman]

Ended up using the following code in B4i and it seem to fix the issue.

msg1 = msg1.Replace(Chr(0),"")

I don't know anything about MQTT and have only glanced at this thread, so take this with a huge grain of salt:

I think this solution might work nicely for this case, but will cause strange errors from time to time. The reason is that (I guess) that the payload can actually contain chr(0) in the correct message. And when you remove them, you corrupt the message. I imagine there is some padding happening, which is why there are chr(0) added to the end of the message. Which means that instead of just doing a search-and-replace, you'd want to trim those chars away on the end part so that's only what's removed. Not close to my Windows computer so I can't check what support we have for this with built-in commands, worst case you'd need to do a simple loop yourself.

 

[emexes]

I don't know anything about MQTT and have only glanced at this thread, so take this with a huge grain of salt:

No worries. All valid points from a broad perspective, but in this particular case I think we have them covered:

the payload can actually contain chr(0) in the correct message.

The payload here is Base64Encoded, so will contain only ASCII printable characters.

trim those chars away on the end part so that's only what's removed ... do a simple loop

ie, workaround at top of post #13 (albeit: no loop ;-) but the .Replace one-liner is clearer again, hence my mild embarrassment...

 

[OliverA]

Righto, the problem is that the B4I Cipher.Decrypt routine is not removing trailing padding of Chr(0).

Somebody here needs to file a bug report. Interesting that all this time, no one noticed. I was going to see if using the Decrypt2 method would reproduce the issue, but iOS is not very user friendly when it comes to encryption. In this case, I could not figure out how to do a salted SHA1 password hash of 1024 iterations. Stuff is so much simpler in B4A/B4J due to Java's encryption framework and the great folks that wrote BouncyCastle. Source for B4J B4XEncryption can be found here: https://github.com/AnywhereSoftware...nywheresoftware/b4x/object/B4XEncryption.java