Android Question Usable encryption between B4A and ESP8266

[max123]

Hi all,

for my 3D printing host app I need to encrypt some data while send over network, then decrypt on server side that receive over the network and send to the 3D Printer over USB.
For this purpose I've used a Raspberry PI Zero W that acts as WiFi-Serial bridge to remotely control a 3D printer.

Because I had necessity to adapt to ESP8266 and ESP32 (instead of Raspberry), I never used B4XEncryption and tried to use a sort of simple algorithm XORing all strings I send.
This worked well between my Android SmartPhone and Raspberry (B4A <-> B4J), but now that I converted the server side to run on ESP8266 (In C++ on Arduino IDE) instead of Raspberry it won't work correctly. I've found a lots of problems because different characters encoding UTF8 vs ASCII (or extended ASCII).

Some strings are decrypted good and some others are incomplete and/or not correct decrypted.

The algorithm work well from Android to Android using B4A, I even tried it on B4J and seem to work well too.
The same work well from ESP8266 to another ESP8266 or ESP32 or dencrypt/decrypt on the same sketch.

But when I try it between different platforms eg. B4A <-> ESP8266, as already explained, sometime work and sometime not.

Note that because I use XOR there are no function to Encrypt and function to Decrypt, just call the function will encrypt the string, calling it again will decrypt.
This function is pretty fast too, it encrypt/decrypt about 3.500.000 characters every second on my old Dualcore PC, but won't work between different platforms.
On ESP8266 with CPU clocked @ 160Mhz it require about 3 microseconds every character to encrypt/decrypt, so very fast too...

On B4X side I use this:

Sub XorMessage (Text As String, Key As String) As String
    Dim out As StringBuilder
    out.Initialize
 
    Dim msglen As Int = Text.Length
    Dim keylen As Int = Key.Length

    Dim n1 As Int
    Dim n2 As Int

    For i = 0 To msglen-1
        n1 = Asc(Text.CharAt(i))
        n2 = Asc(Key.CharAt(i Mod keylen))
        out.Append(Chr(Bit.Xor(n1, n2)))
    Next
    Return out.ToString
End Sub

On Arduino IDE side I use this:

String XorMessage (String Msg, String Key) {
   String tmpMsg = "";
   String tmpKey = "";

   if (fromUTF8) {
      tmpMsg = utf8ascii (Msg);
      tmpKey = utf8ascii (Key);
   } else {
      tmpMsg = Msg;
      tmpKey = Key;
   }

   const unsigned int msglen = tmpMsg.length();
   const unsigned int keylen = tmpKey.length();
   String out = "";

   for (unsigned int i = 0; i < msglen; i++) {
      out += (char) (tmpMsg[i] ^ tmpKey[i % keylen]);
   }
   return out;
}

Note that I've used String object, but the right way will be tu use a char array. Anyway Ive used it in conjuntion with Arduino utf8ascii functions that convert UTF8 to extended ASCII by check multiple bytes.
Arduino utf8ascii

In Arduino String object (as it will be) the string length() functions return the byte length, so any 'special' charater outside ASCII code return the byte length, eg:

String s = "è";
// This return 2 because 'è' characters is 2 bytes encoded in UTF8
Serial.println(s.length());

will return 2 and not 1. It can be from 1 to 4 bytes depending of the character.

To the B4X side I actually manage default UTF8 encode, maybe manage it with String.getBytes("ASCII") instead of String.getBytes("UTF-8") ?
But in this code I never used String.getBytes at all.... tried it with others encryption algorithms but not here in these functions.

Please guys, is there a simple way to do this?
Many thanks

 

[agraham]

I think you are making the common mistake of confusing characters with bytes. B4X characters as 16 bit entities whereas you seem to be assuming they are 8 bits. You probably need to explicitly use byte values in both Arduino and B4X and make sure the byte representations of the strings are the same on both platforms.

 

[max123]

Many thanks @agraham,

teorically first 127 characters (7 bits) of UTF8 are same that ASCII.

I searched to do it, but how? to convert String to Bytes I need String.getBytes(charset). is this the way?
Using as charset ASCII ? If I use ASCII it will truncate all characters highter than 127 ? is not possible to have 256 chars as extended ASCII ?
anotther one is that byte in B4X side is -127 to 127 like in java, right ?

Or maybe using ByteBuilder?

Many thanks for your interest.
Massimo

 

[agraham]

Use the ByteConverter library on B4X and keep with bytes on Arduino.

 

[max123]

I never used your ByteConverter library, what method I need to use...
StringToBytes and StringFromBytes ?

It is possible to use extended ASCII whithout truncate any value to 7 bits?
The Arduino internally can handle extended ASCII and UTF8 too but maybe I
need to create a character table, not sure for this.

I do not use special characters, but maybe accented letters like èùà etc or some symbols,
as I can see these are inside 128-256 characters of extended ASCII.

Please can you provide a simple example on how I can convert
my simple function using ByteConverter?

Many thanks

 

[agraham]

StringToBytes and StringFromBytes ?

Probably

It is possible to use extended ASCII whithout truncate any value to 7 bits?

Of course, using the correct code page.

Please can you provide a simple example on how I can convert my simple function using ByteConverter?

Sorry, you will have to work that out for yourself - it will be a good learning process.

 

[max123]

So something like this can work?

I've tried it on B4J for convenience and on the log both seem to have same results. Both returns values from 0 to 255, I can see accented and special characters 128-255 on the log.

I searched on the web for a right encoder (I think it is right but not sure) anyway is the standard ISO Latin.

What is best of these two functions for your experience?

    Dim original As String = "Questa è una stringa relativamente lunga da usare come test per l'algoritmo di encrypt: Numeri: 0123456789 Caratteri: !?è+òàù=()È+ÒÀÙ+-()fine"
'    Dim original As String = "Lettere: abcdefghilmnopqrstuvzxyj ABCDEFGHILMNOPQRSTUVZXYJ  Numeri: 0123456789 Caratteri: !?è+òàù=()È+ÒÀÙ+-()fine"
 
    Log(" Original: [" & original & "]   Len: " & original.Length)

    Encrypted = XorMessage(original, ENCRYPTION_KEY)
    Log("Encrypted: [" & Encrypted & "]   Len: " & Encrypted.Length)
    Decrypted = XorMessage(Encrypted, ENCRYPTION_KEY)
    Log("Decrypted: [" & Decrypted & "]   Len: " & Decrypted.Length)
 
    Log("-------------------------------------------")
    If Decrypted = original Then
        Log("PASSED")
    Else
        Log("NOT PASSED")
    End If
    Log("-------------------------------------------")
 
    For i = 0 To Decrypted.Length-1
        Log(i & "[" & Decrypted.CharAt(i) & "]   [" & Asc(Decrypted.CharAt(i)) & "]")
        Sleep(200)
    Next

Sub XorMessage1 (Text As String, Key As String) As String
    Dim out As StringBuilder
    out.Initialize
 
    Dim bc As ByteConverter
    ' US-ASCII only 0-127, ISO-8859-1 Latin 0-255
    Dim TextBytes() As Byte = bc.StringToBytes(Text, "ISO-8859-1")
    Dim KeyBytes() As Byte = bc.StringToBytes(Key, "ISO-8859-1")
 
    Dim msglen As Int = TextBytes.Length
    Dim keylen As Int = KeyBytes.Length

    Dim n1 As Int
    Dim n2 As Int

    For i = 0 To msglen-1
        n1 = ToUnsigned(TextBytes(i))
        n2 = ToUnsigned(KeyBytes(i Mod keylen))
        out.Append(Chr(Bit.Xor(n1, n2)))
    Next
    Return out.ToString
End Sub

Sub ToUnsigned(b As Byte) As Int
    Return Bit.And(b, 0xFF)
End Sub

Sub XorMessage2 (Text As String, Key As String) As String
    Dim bc As ByteConverter
    ' US-ASCII only 0-127, ISO-8859-1 Latin 0-255
    Dim TextBytes() As Byte = bc.StringToBytes(Text, "ISO-8859-1")
    Dim KeyBytes() As Byte = bc.StringToBytes(Key, "ISO-8859-1")
 
    Dim msglen As Int = TextBytes.Length
    Dim keylen As Int = KeyBytes.Length

    Dim byteOut(msglen) As Byte
 
    Dim n1 As Byte
    Dim n2 As Byte

    For i = 0 To msglen-1
        n1 = ToUnsigned(TextBytes(i))
        n2 = ToUnsigned(KeyBytes(i Mod keylen))
        byteOut(i) = Bit.Xor(n1, n2)
    Next
    Return bc.StringFromBytes(byteOut, "ISO-8859-1")
End Sub

Many thanks

 

[agraham]

XorMessage2 looks about right to me. I'm not sure that you need the Unsigned Sub though.

a right encoder (I think it is right but not sure)

The exact encoding doesn't matter as long as it has the characters that you require and you use the same at both ends.

 

[max123]

Ok many thanks, now I edit a counterpart C++ function to match this one....

And then I will try on my app to know if it is working between platforms, I think yes if it use just bytes.

Because I'm maniac in my project test for the encryption already I added cycles to process billion of characters measuring precisely the speed of function execution, so I will test both... at this point I'm curious to know what of both is faster.

I'm not sure that you need the Unsigned Sub though.

Yes, really as you said no need for this because already are signed bytes -128 to 127 and have no sense AND it witn 255.
It have sense in the XorMessage1 because use integer values and need it.

Many thanks

 

[max123]

Hi @agraham,

I've tried this on Arduino IDE side but with strange results:

  String ASCII = "", ISO8859 = "";
   for (int i = 32; i < 256; i++) { // Show full set of ASCII characters ( > 127 will be ignored)
      ASCII += (char)i;
   }
   Serial.printf ("  ASCII: [%s]\nLen: %d\n\n", ASCII.c_str(), ASCII.length());

   for (int i = 32; i < 256; i++) { // Show full set of ISO-8859-1 Latin characters (0-255)
      ISO8859 += GetUnicodeChar (i);
   }
   Serial.printf ("ISO8859: [%s]\nLen: %d\n\n", ISO8859.c_str(), ISO8859.length());

   String myString = "Questa è una lunga stringa 1234567890 ÒÀÈÙòèàù";
   Serial.printf ("  ASCII: [%s]   Len: %d\n\n", myString.c_str(), myString.length());
   myString = GetIsoString (myString);
   Serial.printf ("ISO8859: [%s]   Len: %d\n\n", myString.c_str(), myString.length());

/////////// ISO-8859-1 //////////

String UNICODE[] = {"", "", "", "", "", "", "", "", "", "",           // 128 to 137 (Not used)
                    "", "", "", "", "", "", "", "", "", "",           // 138 to 147 (Not used)
                    "", "", "", "", "", "", "", "", "", "",           // 148 to 157 (Not used)
                    "", "", " ", "¡", "¢", "£", "¤", "¥", "¦", "§",   // 158, 159 (Not used)  (160 to 167 used)
                    "¨", "©", "ª", "«", "¬", "", "®", "¯", "°", "±",  // 168 to 177 (used)
                    "²", "³", "´", "µ", "¶", "·", "¸", "¹", "º", "»", // 178 to 187 (used)
                    "¼", "½", "¾", "¿", "À", "Á", "Â", "Ã", "Ä", "Å", // 188 to 197 (used)
                    "Æ", "Ç", "È", "É", "Ê", "Ë", "Ì", "Í", "Î", "Ï", // 198 to 207 (used)
                    "Ð", "Ñ", "Ò", "Ó", "Ô", "Õ", "Ö", "×", "Ø", "Ù", // 208 to 217 (used)
                    "Ú", "Û", "Ü", "Ý", "Þ", "ß", "à", "á", "â", "ã", // 218 to 227 (used)
                    "ä", "å", "æ", "ç", "è", "é", "ê", "ë", "ì", "í", // 228 to 237 (used)
                    "î", "ï", "ð", "ñ", "ò", "ó", "ô", "õ", "ö", "÷", // 238 to 247 (used)
                    "ø", "ù", "ú", "û", "ü", "ý", "þ", "ÿ"            // 248 to 255 (used)
                   };

// This is not a real char, return (one) character but can be one or more bytes,
// so we use String class to collect, we cannot use char to handle a single character
String GetUnicodeChar (uint8_t tByte) {
   String out = "";
   if (tByte < 128) {
      out += (char) tByte; // Append (Mantain the same ASCII code, cast to char)
   } else {
      out += UNICODE [tByte - 128]; // Append
   }
   return out;
}

String GetIsoString (String Text) {
   String out = "";
   uint8_t CurrentByte;
   unsigned int len = Text.length();
   for (unsigned int i = 0; i < len; i++) {
      CurrentByte = (uint8_t) Text[i];  // Or maybe use charAt ?
      out += GetUnicodeChar (CurrentByte);
   }
   return out;
}

Here the result of this code:

ASCII: [ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮⸮]
Len: 224

ISO8859: [ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~ ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖ×ØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõö÷øùúûüýþÿ]
Len: 285

ASCII: [Questa è una lunga stringa 1234567890 ÒÀÈÙòèàù] Len: 55

ISO8859: [Questa è una lunga stringa 1234567890 ÃÃÃÃòèà ù] Len: 64

I'm a pretty confused now....

If I decode one string to ISO it is right, but this do not match, Arduino String class seem to use UTF8, even Arduino Serial monitor support it, I can copy special characters eg. from the web and past on it or in the editor. To make a table I just do copy/paste from the web and that worked.

But now I cannot know how to proceed I need to recreate original string sent (as bytes) from B4X

Please, do you know what is wrong here ?

Many thanks

 

[agraham]

I've no idea what you mean by 'strange result' but it is almost certain to be an encoding problem converting bytes to characters on the Arduino. For example this line is still confusing bytes and characters.

     CurrentByte = (uint8_t) Text[i];  // Or maybe use charAt ?

You are assuming the Arduino native coding is ISO-8859-1 and it almost certainly isn't. In the same way as you used StringToBytes in B4X to get ISO-8859-1 byte values from native UTF16 characters you need to do the converse on the Arduino to get whatever the native coding characters are from ISO-8859-1 byte values.

 

[max123]

So now I need to convert back ISO-8859-1 to UTF-8 ?

Many thanks

 

[agraham]

So now I need to convert back ISO-8859-1 to UTF-8

Yes

with utf8ascii posted on post #1 ?

Maybe, I don't know enough about Arduino.

 

[max123]

I will try it....

Is not clear to me how Arduino IDE works, the IDE support UTF8 and Unicode, but string functions do not see characters > 127, so pure ASCII.

Many thanks