Android Tutorial LockBox3 (Delphi) AES encryption exchange with B4A/B4J

OliverA

Expert
Licensed User
Part 1: The key

This is just a write up of my findings regarding the exchange of encrypted data between B4A/B4J and LockBox3 (Delphi). This write up focuses just on LockBox3's implementation of AES (128, 192, and 256 bit) cipher and the block modes ECB and CBC. On the Delphi side, before proceeding to exchange string data with the outside world, the programmer needs to ensure that all string encoding is done via UTF8.

For a quick overview of AES, feel free to read @KMatle's write up (https://www.b4x.com/android/forum/threads/b4x-aes-encryption-lessons-learned-best-practice.97927/).

As mentioned in @KMatle's write up, the key size of AES is either 128, 192, or 256. That translates into password lengths of 16, 24, or 32 bytes. The way that LockBox generates the key is as follows:

1) The password string is converted into bytes, using the appropriate encoding. For our purposes, the encoding needs to be UTF8.

2) If the number of bytes is equal to or greater than the password length required for the appropriate key size, then use the password, truncating it to the proper size if necessary.

3) If the number of bytes is less than the key size, create a SHA1 hash out of the password bytes. This hash is 20 bytes long. If the hash is long enough for the key size in question, use it, truncating it to the proper size. Otherwise, keep concatenating the hash until the proper key size is produced.

B4X:
Private Sub MakeKeyBytes(keySize As Int, password As String) As Byte()
   'If the password is keySize / 8 bytes or greater in length, just truncate password to proper byte size
   'If password is less than keySize /8 bytes in length, create a SHA1 digest of the password
   ' (20 bytes), and concatenate the digest and truncate to proper byte size.

   Dim bc As ByteConverter
   Dim keyBytes(keySize / 8) As Byte
   Dim passwordBytes() As Byte = password.GetBytes("UTF8")

   If passwordBytes.Length >= keyBytes.Length Then
       bc.ArrayCopy(passwordBytes, 0, keyBytes, 0, keyBytes.Length)
   Else
       Dim md As MessageDigest
       Dim hash() As Byte = md.GetMessageDigest(passwordBytes, "SHA1")
       Dim offset As Int = 0
       Do While offset < keyBytes.Length
           If offset + hash.Length < keyBytes.Length Then
               bc.ArrayCopy(hash, 0, keyBytes, offset, hash.Length)
           Else
               bc.ArrayCopy(hash, 0, keyBytes, offset, keyBytes.Length - offset)
           End If
           offset = offset + hash.Length
       Loop
   End If

   Return keyBytes
End Sub
Note: The above code (and future code) uses @agraham's ByteConverter library (https://www.b4x.com/android/forum/threads/byteconverter-library.6787/#content).
 

OliverA

Expert
Licensed User
Part 2: ECB

AES is a block cipher and encrypts data one block at a time. There are various ways to process these blocks and here I will first address ECB and then CBC as it is implemented by LockBox3.

In AES, each block is 16 bytes long. Depending on the block mode used, data that does not fit nicely into 16 byte chunks may need to be padded. There are various ways to pad a block (see https://en.wikipedia.org/wiki/Padding_(cryptography)#Byte_padding). In the case of ECB, LockBox3 has settled for ISOIEC 7816-4. From my rudimentary knowledge of encryption with B4X/Java, the only padding schemes I was aware of were no padding and PKCS5/PKCS7 padding. Because of this, in my first attempt to tackle the issue of padding (see https://www.b4x.com/android/forum/threads/encrypt-and-decrypt-data.91022/#post-575824), I’ve padded the data to be encrypted via code. Even though this seemed to work and received some likes, I’ve should not have padded the data myself. For one, the code just pads the data, even if it is not necessary (a 16 byte aligned data receives a 16-byte padded block at the end). This does not break the decryption of the data, but it does not follow the requirements of padding. It also weakened an already weak bock mode (ECB is not a recommended mode to use. See https://crypto.stackexchange.com/questions/20941/why-shouldnt-i-use-ecb-encryption) by always have a deterministic ending to the data. Secondly, the padding is available through the usage of Bouncy Castle Crypto APIs (https://www.bouncycastle.org/). For Android, Bouncy Castle is already available through the OS, whereas B4J just requires one additional .jar file (https://www.bouncycastle.org/download/bcprov-jdk15on-160.jar) and the usage of #AdditionalJar: bcprov-jdk15on-160. To access Java’s/Android’s underlying encryption framework, we make use of the Encryption library provided by @agraham. For encoding/decoding the encrypted information, we will also be using the build in StringUtils/jStringUtils library.

So, let’s code some ECB encryption. For this code, we assume that the variables keySize (As Int), set to either 128, 192, or 256, password (As String), containing the user/program provided password, and plainText (As String), the text to be encrypted, have already been set. The result will be that the variable cipherText will contain our Base64 encoded encrypted data.

B4X:
Dim su As StringUtils
Dim c As Cipher
Dim k As KeyGenerator
Dim cipherText As String = ""
Dim keyBytes() As Byte = MakeKeyBytes(keySize, password)
Dim dataBytes() As Byte = plainText.GetBytes("UTF8")

k.Initialize("AES")
k.KeyFromBytes(keyBytes)

c.Initialize("AES/ECB/ISO7816-4PADDING")
Dim cipherBytes() As Byte = c.Encrypt(dataBytes, k.Key, False)

cipherText = su.EncodeBase64(cipherBytes)
Now cipherText contains a proper padded output, without resorting to manually adding padding.

For decrypting, plainText will be generated and cipherText contains the Base64 code to be decrypted.

B4X:
Dim su As StringUtils
Dim c As Cipher
Dim k As KeyGenerator
Dim plainText As String = ""
Dim keyBytes() As Byte = MakeKeyBytes(keySize, password)
Dim dataBytes() As Byte = su.DecodeBase64(cipherText)

k.Initialize("AES")
k.KeyFromBytes(keyBytes)

c.Initialize("AES/ECB/ISO7816-4PADDING")
Dim plainTextBytes() As Byte = c.Decrypt(dataBytes, k.Key, False)

plainText = BytesToString(plainTextBytes, 0, plainTextBytes.Length, "UTF8")
This time, no manual removing of padding.

Note: The oldest version of Android that I have on a mobile is 5.1. Older versions may have a Bouncy Castle implementation that does not contain the support for ISO7816-4PADDING. If that happens to be the case, let me know in the comments and I’ll update this post with steps on how to include Spongy Castle in your Android project.
 

OliverA

Expert
Licensed User
Part 3: CBC

2019/08/26: As pointed out by @PassionDEV, the below implementation works incorrectly in cases where 1) the plaintext or ciphertext byte array is greater than 16 bytes and 2) the array of bytes is not a multiple of 16. I'll add part 4 that will address (and solve) this case.

Onto CBC block mode. Unlike ECB mode, CBC mode uses an initialization vector (IV). LockBox3 manages the IV’s automatically for you. First, LockBox3 will create a 8 byte random nonce (https://en.wikipedia.org/wiki/Cryptographic_nonce) and us the nonce for the first 8 bytes of the 16 byte IV (IV’s are 16 byte long – see @KMatle's post). Once LockBox3 is done encrypting your data, it will prepend the encrypted data with the nonce (only the 8 byte nonce) and then Base64 encode all that to produce the cipher text. LockBox3, being Lockbox3, then proceeds to not use any padding with CBC, but use something called ciphertext stealing instead (CTS) (https://en.wikipedia.org/wiki/Ciphertext_stealing).

When I first tackled this issue, I could not find any out of the box support for ciphertext stealing. Not only that, but for data that was smaller than one block (16 bytes), LockBox3 would switch to the block mode CFB (with a length of 8 bit). The first solution that I implemented used the same padding as the initial ECB code (this is where I realized that my ECB padding was incorrect). This solved two issues. One, the smallest amount of data I had to encrypt was 16 bytes and therefore I would not have to worry about CFB. Two, since the padded data is evenly divisible by 16, I did not have to worry about CTS and I could just use AES/CBC/NoPadding for encryption and decryption. The problem was that on the Delphi side, one would have to add the padding before encrypting and remove the padding after decrypting. That just didn’t seem ideal. After doing some more research, I came across methods in Bouncy Castle that I could call via JavaObject that would let me implement proper CBC/CTS and handle CFB 8bit. Life was great. Except it still looked hacky to me. Being able to use a transformation string (the “AES/CBC/NoPadding”) would be so much simpler. It took me reading some of the code of Bouncy Castle to realize that it could handle CTS and CFB as required. It was at that time that I also discovered the padding option for the ECB block mode. And if you don’t want to look at code, this page (https://www.bouncycastle.org/specifications.html) shows all the things you can use in your transformation string as it pertains to Bouncy Castle (a page that either never pulled up in my searches or I just happen to conveniently ignore). It turns out that for CTS, the transformation string is “AES/CTS/NoPadding” and for the CFB it is “AES/CFB8/NoPadding”.

Now to some coding. As with ECB encryption, we assume that keySize, password, and plainText have been set. SecureRandom is part of the Encryption library and allows us to create a “cryptographically strong random number” (https://docs.oracle.com/javase/7/docs/api/java/security/SecureRandom.html) that will be used for the nonce.

B4X:
Dim su As StringUtils
Dim c As Cipher
Dim k As KeyGenerator
Dim sr As SecureRandom
Dim bc As ByteConverter
Dim cipherText As String = ""
Dim keyBytes() As Byte = MakeKeyBytes(keySize, password)
Dim dataBytes() As Byte = plainText.GetBytes("UTF8")

k.Initialize("AES")
k.KeyFromBytes(keyBytes)

'Autogenerate nonce and insert into IV
Dim nonce(8) As Byte
sr.GetRandomBytes(nonce)
Dim IV(16) As Byte
bc.ArrayCopy(nonce, 0, IV, 0, 8)


If dataBytes.Length > 15 Then
            c.Initialize("AES/CTS/NoPadding")
Else
           c.Initialize("AES/CFB8/NoPadding")
End If
c.InitialisationVector = IV
Dim encryptedBytes() as Byte = c.Encrypt(dataBytes, k.Key, True)

'Create result byte array, consisting of nonce and encrypted data
Dim cipherTextBytes(encryptedBytes.Length + 8) As Byte
bc.ArrayCopy(nonce, 0, cipherTextBytes, 0, 8)
bc.ArrayCopy(encryptedBytes, 0, cipherTextBytes, 8, encryptedBytes.Length)

cipherText = su.EncodeBase64(cipherTextBytes)
When decrypting, the nonce and data need to be first extracted from the encrypted string. The nonce is than used to set up the IV. As with the encryption, the type of block mode to use is determined by the length of the encrypted data.

B4X:
Dim su As StringUtils
Dim c As Cipher
Dim k As KeyGenerator
Dim sr As SecureRandom
Dim bc As ByteConverter
Dim plainText As String = ""
Dim keyBytes() As Byte = MakeKeyBytes(keySize, password)

'Retrieve nonce and data to decrypt from ciphertext byte array
Dim cipherTextBytes() As Byte = su.DecodeBase64(cipherText)
Dim nonce(8) As Byte
bc.ArrayCopy(cipherTextBytes, 0, nonce, 0, 8)
Dim dataBytes(cipherTextBytes.Length - 8) As Byte
bc.ArrayCopy(cipherTextBytes, 8, dataBytes, 0, cipherTextBytes.Length - 8)

'Create IV from nonce
Dim IV(16) As Byte
bc.ArrayCopy(nonce, 0, IV, 0, 8)

If dataBytes.Length > 15 Then
            c.Initialize("AES/CTS/NoPadding")
Else
           c.Initialize("AES/CFB8/NoPadding")
End If
c.InitialisationVector = IV
Dim plainTextBytes() As Byte = c.Decrypt(dataBytes, k.Key, True)

plainText = BytesToString(plainTextBytes, 0, plainTextBytes.Length, "UTF8")
Hopefully this write up will be helpful to anyone that needs to implement/cooperate with another platform’s encryption scheme. As you see, the underlying encryption frameworks of Java/Android are quite powerful. The B4J side needs some help from Bouncy Castle and
Android may need Spongy Castle to enhance their respective capabilities. Thanks to @agraham's library that exposes that framework in a programmer friendly fashion, I was able to re-implement LockBox3’s AES ECB and CBC modes without resorting to hacks, a ton of JavaObject usage and/or the reliance on in-line Java. I’ll post my class implementation of this in the libraries/classes forum (https://www.b4x.com/android/forum/t...-ecb-cbc-encrypting-decrypting.97962/#content).
 
Last edited:

OliverA

Expert
Licensed User
Part 4: CBC (Lockbox3's version) implemented correctly

Lockbox3 distinguishes between 4 types of plaintext / ciphertext byte arrays

1) Zero length/empty message: The output should be an empty result. This is covered in the published library
2) Short messages: A message that has an underlying byte array of less then 16 bytes. This is covered in published library/code above
3) Round messages: A message that has an underlying byte array of 16 or more bytes and whose length is a multiple of 16. This is covered in the published library/code above
4) Sharp message: A message that has an underlying byte array of more than 16 bytes and whose length is not a multiple of 16. This is where everything goes wrong. For these types of messages, Lockbox3 uses ciphertext stealing. In the crypto libraries for Java (B4J/B4A), this refers to the CTS mode of a cipher. If you look at the code above, I use CTS mode for all messages larger than 15 bytes. For messages that are a multiple of 16 in length, CTS is just plain CBC and does not perform any ciphertext stealing. So with the code above (in the case of encryption)
B4X:
If dataBytes.Length > 15 Then
            c.Initialize("AES/CTS/NoPadding")
Else
           c.Initialize("AES/CFB8/NoPadding")
End If
c.InitialisationVector = IV
Dim encryptedBytes() as Byte = c.Encrypt(dataBytes, k.Key, True)
I thought I had cases 2 through 4 covered, with the CFB8 mode covering case 2 and the CTS mode covering case 3 and 4 (case #1 is a quick check on the incoming ciphertext/plaintext to see if it contains anything and returning an empty string in case of no content). The problem is, that for sharp messages, everything seemed to encrypt/decrypt correctly except for the last two blocks. It is the last two blocks that ciphertext stealing concentrates upon.

So now what? First, lets look at the steps performed upon the last two blocks in a "normal" ciphertext stealing process (source: https://en.wikipedia.org/wiki/Ciphertext_stealing) (BTW, everything up to the last two blocks uses standard CBC for decryption/encryption):

For decryption:

c = ciphertext bytes
p = plaintext bytes

N = the last block
N - 1 = the second to last block
N - 2 = the third to last block

Prep work:
If c(N-2) exists, set IV to c(N-2), otherwise the IV is the IV passed to the algorithm

Step A:
Decrypt c(N-1) using ECB mode with given key

Step B:
1) Take c(N) and fill in the remaining bytes from the result generated in step A above to create a 16 byte c(N) block. For example, if c(N) is 15 bytes long, the fill in its 16th byte from the 16th byte of step A result
2) Decrypt c(N) using ECB mode with the given key and XOR the result with IV, generating p(N-1)

Step C:
1) XOR the result from step A with the result from step B1, generating p(N)
2) Trim p(N) to original length of c(N)

Final steps:

Decrypt (ciphertext - last two blocks) using CBC mode with given IV and password
Append p(N-1) and then p(N).

So how does Lockbox3 differ? I looked at the source code, but alas, I could not decipher the Pascal (Delphi) code. Luckily, I found an article on stackoverflow: https://stackoverflow.com/a/12330169.

LockBox3 CTS decryption:

Prep work:
Unchanged

Step A:
1) Decrypt c(N-1) using ECB mode with given key
2) XOR result with IV, generating p(N) (Notice Lockbox uses the same IV for both XOR's)

Step B:
1) Take c(N) and fill in the remaining bytes from the result generated in step A2 above to create a 16 byte c(N) block. For example, if c(N) is 15 bytes long, the fill in its 16th byte from the 16th byte of step A2 result
2) Decrypt c(N) using ECB mode with the given key and XOR the result with IV, generating p(N-1)

Step C:
1)
2) Trim p(N) to original length of c(N)

Final steps:

Decrypt (ciphertext - last two blocks) using CBC mode with given IV and password
Append p(N-1) and then p(N).

So, a simple
B4X:
If dataBytes.Length > 15 Then
            c.Initialize("AES/CTS/NoPadding")
Else
           c.Initialize("AES/CFB8/NoPadding")
End If
c.InitialisationVector = IV
Dim plainTextBytes() As Byte = c.Decrypt(dataBytes, k.Key, True)

plainText = BytesToString(plainTextBytes, 0, plainTextBytes.Length, "UTF8")
becomes
B4X:
           Dim decryptedBytes() As Byte

           If dataBytes.Length > 15 Then
               Dim fullBlocks As Int = dataBytes.Length / 16
               Dim lastBlockSize As Int = dataBytes.Length Mod 16
               Dim numberOfBlocks As Int = fullBlocks
               If lastBlockSize > 0 Then numberOfBlocks = numberOfBlocks + 1
               cCBC.InitialisationVector = IV
               If fullBlocks = numberOfBlocks Then
                   'We have a round message
                   decryptedBytes = cCBC.Decrypt(dataBytes, k.Key, True)
               Else
                   'We have a sharp message
                   Dim bb As B4XBytesBuilder
                   bb.Initialize

                   'Everything but the last full block and it's trailing partial block can be decrypted via standard CBC
                   If fullBlocks > 1 Then
                       Dim frontCipherData(dataBytes.Length - 16 - lastBlockSize) As Byte
                       bc.ArrayCopy(dataBytes, 0, frontCipherData, 0, dataBytes.Length - 16 - lastBlockSize)
                       Dim frontCipherDecrypted() As Byte = cCBC.Decrypt(frontCipherData, k.Key, True)
                       bb.Append(frontCipherDecrypted)
                       'IV is the last block of "frontCipherData"
                       bc.ArrayCopy(frontCipherData, frontCipherData.Length - 16, IV, 0, 16)
                   End If

                   'Take 2nd to last block of ciphertext data, ECB decrypt and XOR with IV
                   Dim secondToLastBlock(16) As Byte
                   bc.ArrayCopy(dataBytes, dataBytes.Length - 16 - lastBlockSize, secondToLastBlock, 0, 16)
                   Dim lastBlockDecrypted() As Byte = cECB.Decrypt(secondToLastBlock, k.Key, False)
                   For x = 0 To lastBlockDecrypted.Length - 1
                       lastBlockDecrypted(x) = Bit.Xor(lastBlockDecrypted(x), IV(x))
                   Next

                   'Take last block of ciphertext data, fill remaining bytes with corresponding bytes (in position) from lastBlockDecrypted created above,
                   ' ECB decrypt and XOR with IV (note: IV does not change!)
                   Dim lastBlock(16) As Byte
                   bc.ArrayCopy(dataBytes, dataBytes.Length - lastBlockSize, lastBlock, 0, lastBlockSize)
                   For x = lastBlockSize To lastBlock.Length - 1
                       lastBlock(x) = lastBlockDecrypted(x)
                   Next
                   Dim secondToLastBlockDecrypted(16) As Byte = cECB.Decrypt(lastBlock, k.Key, False)
                   For x = 0 To secondToLastBlockDecrypted.Length - 1
                       secondToLastBlockDecrypted(x) = Bit.Xor(secondToLastBlockDecrypted(x), IV(x))
                   Next

                   'Add the last two blocks and generate plaintext bytes
                   bb.Append(secondToLastBlockDecrypted)
                   'Trim last block to appropriate size
                   Dim lastBlockDecryptedSized(lastBlockSize) As Byte
                   bc.ArrayCopy(lastBlockDecrypted, 0, lastBlockDecryptedSized, 0, lastBlockSize)
                   bb.Append(lastBlockDecryptedSized)
                   decryptedBytes = bb.ToArray
               End If
           Else
               'We have a short message
               cCFB.InitialisationVector = IV
               decryptedBytes = cCFB.Decrypt(dataBytes, k.Key, True)
           End If
           plainText = BytesToString(decryptedBytes, 0, decryptedBytes.Length, "UTF8")
So please, for anyone thinking about creating their own encryption library, don't. Sooner or later, someone wants to interoperate with your library and then the same person or someone else has to figure out what the heck you did!
 
Top