Android Question .Net GZip Compression and B4A Decompression Problem

[Ohanian]

Hi,

i'm trying to compress/decompress data from an asp.net web service to a b4a client.

here's the .net part :

Context.Response.Expires = -1
'Context.Response.ContentType = "application/x-gzip"
Context.Response.ContentType = "application/json"
Context.Response.ContentEncoding = System.Text.Encoding.UTF8
Context.Response.Write(GZIPCompress("Hello, This is a sample text."))

 Function GZipCompress(ByVal s As String) As String
        Dim bytes = Encoding.Unicode.GetBytes(s)
        Using msi = New MemoryStream(bytes)
            Using mso = New MemoryStream()
                Using gs = New GZipStream(mso, CompressionMode.Compress)
                    msi.CopyTo(gs)
                End Using
                'Return Convert.ToBase64String(mso.ToArray())
                Return Encoding.UTF8.GetString(mso.ToArray)
            End Using
        End Using
    End Function

and b4a part :

Dim sData As String   
        Dim cs As CompressedStreams
        Dim In As InputStream
        In = cs.WrapInputStream(Job.GetInputStream, "gzip")       
        Dim Reader As TextReader
        Reader.Initialize(In)
        sData = Reader.ReadAll
       
        Log(sData)

but got errors like these :
java.io.IOException: unknown format (magic number 7364)
java.io.IOException: unknown format (magic number 3448)
java.io.IOException: unknown format (magic number ef1f)

any suggestion?

 

[Erel]

any suggestion?

Use B4J to implement the server side. It will be much simpler.

This line is wrong:

Return Encoding.UTF8.GetString(mso.ToArray)

You cannot treat raw bytes as a UTF8 string. You need to either send the raw bytes or convert them to a string with base 64 encoding.

 

[Ohanian]

Hi,

Use B4J to implement the server side. It will be much simpler.

it's a part of a bigger project, so it's not possible to change the server side.

I've got 3 different results from .net side!

                Dim sData As String = String.Empty
                sData = GZIPCompress("Hello, This is a sample text.")
                sData = CompressString("Hello, This is a sample text.")
                sData = Zip("Hello, This is a sample text.")

 Function GZipCompress(ByVal s As String) As String
        Dim bytes = Encoding.Unicode.GetBytes(s)
        Using msi = New MemoryStream(bytes)
            Using mso = New MemoryStream()
                Using gs = New GZipStream(mso, CompressionMode.Compress)
                    msi.CopyTo(gs)
                End Using
                Return Convert.ToBase64String(mso.ToArray())
                'Return Encoding.UTF8.GetString(mso.ToArray)
            End Using
        End Using
    End Function

 Function CompressString(ByVal text As String) As String
        Dim buffer__1 As Byte() = Encoding.UTF8.GetBytes(text)
        Dim memoryStream = New MemoryStream()
        Using gZipStream = New GZipStream(memoryStream, CompressionMode.Compress, True)
            gZipStream.Write(buffer__1, 0, buffer__1.Length)
        End Using

        memoryStream.Position = 0

        Dim compressedData = New Byte(memoryStream.Length - 1) {}
        memoryStream.Read(compressedData, 0, compressedData.Length)

        Dim gZipBuffer = New Byte(compressedData.Length + 3) {}
        Buffer.BlockCopy(compressedData, 0, gZipBuffer, 4, compressedData.Length)
        Buffer.BlockCopy(BitConverter.GetBytes(buffer__1.Length), 0, gZipBuffer, 0, 4)
        Return Convert.ToBase64String(gZipBuffer)
    End Function

 Public Shared Function Zip(ByVal text As String) As String
        Dim buffer As Byte() = System.Text.Encoding.Unicode.GetBytes(text)
        Dim ms As New MemoryStream()
        Using zipStream As New System.IO.Compression.GZipStream(ms, System.IO.Compression.CompressionMode.Compress, True)
            zipStream.Write(buffer, 0, buffer.Length)
        End Using

        ms.Position = 0
        Dim outStream As New MemoryStream()

        Dim compressed As Byte() = New Byte(ms.Length - 1) {}
        ms.Read(compressed, 0, compressed.Length)

        Dim gzBuffer As Byte() = New Byte(compressed.Length + 3) {}
        System.Buffer.BlockCopy(compressed, 0, gzBuffer, 4, compressed.Length)
        System.Buffer.BlockCopy(BitConverter.GetBytes(buffer.Length), 0, gzBuffer, 0, 4)
        Return Convert.ToBase64String(gzBuffer)
    End Function

but none of them are decompressed in b4a!

Dim sData As String   
    Dim cs As CompressedStreams
    Dim decompressed() As Byte
   
    decompressed = cs.DecompressBytes("H4sIAAAAAAAEAPNgSGXIAcJ8Bh0GBYYQhgyGTIZiIAtCJgJxMZDMZSgAqkkF8kqAZAWQ1GMAAANbXTE6AAAA".GetBytes("UTF8"), "gzip")
    decompressed = cs.DecompressBytes("HQAAAB+LCAAAAAAABADzSM3JyddRCMnILFYAokSF4sTcgpxUhZLUihI9AHKPbswdAAAA".GetBytes("UTF8"), "gzip")
    decompressed = cs.DecompressBytes("OgAAAB+LCAAAAAAABADzYEhlyAHCfAYdBgWGEIYMhkyGYiALQiYCcTGQzGUoAKpJBfJKgGQFkNRjAAADW10xOgAAAA==".GetBytes("UTF8"), "gzip")
   
    sData = BytesToString(decompressed,0, decompressed.Length, "UTF8")
    Log(sData)

but got errors like this :
java.io.IOException: unknown format (magic number 674f)

 

[Erel]

You need to use StringUtils to convert the base64 string to bytes.

 

[Ohanian]

You need to use StringUtils to convert the base64 string to bytes.

the same errors :

Dim su As StringUtils
    Try       
        decompressed = cs.DecompressBytes(su.DecodeBase64("H4sIAAAAAAAEAPNgSGXIAcJ8Bh0GBYYQhgyGTIZiIAtCJgJxMZDMZSgAqkkF8kqAZAWQ1GMAAANbXTE6AAAA"), "gzip")
        sData = BytesToString(decompressed,0, decompressed.Length, "UTF8")
        Log(sData)       
    Catch
        Log(LastException)
    End Try
    Try
        decompressed = cs.DecompressBytes(su.DecodeBase64("HQAAAB+LCAAAAAAABADzSM3JyddRCMnILFYAokSF4sTcgpxUhZLUihI9AHKPbswdAAAA"), "gzip")
        sData = BytesToString(decompressed,0, decompressed.Length, "UTF8")
        Log(sData)       
    Catch
        Log(LastException)
    End Try
    Try
        decompressed = cs.DecompressBytes(su.DecodeBase64("OgAAAB+LCAAAAAAABADzYEhlyAHCfAYdBgWGEIYMhkyGYiALQiYCcTGQzGUoAKpJBfJKgGQFkNRjAAADW10xOgAAAA=="), "gzip")
        sData = BytesToString(decompressed,0, decompressed.Length, "UTF8")
        Log(sData)       
    Catch
        Log(LastException)
    End Try

(IOException) java.io.IOException: unknown format (magic number 48)
(IOException) java.io.IOException: unknown format (magic number 1d)
(IOException) java.io.IOException: unknown format (magic number 3a)

Can you confirm which function on the .net side is ok so I can focus on the b4a code?

 

[Erel]

GZipCompress looks correct (use UTF 8 instead of the ambiguous Unicode encoding).

Note that it might actually be a zlib algorithm instead of gzip.

 

[Ohanian]

Got no result, any other suggestion or approach?

related posts on stackoverflow :

http://stackoverflow.com/questions/6717165/how-can-i-zip-and-unzip-a-string-using-gzipoutputstream-that-is-compatible-with

 

[Erel]

Based on this link the .Net implementation adds 4 bytes (maybe with the length I haven't read it all).

Unless the text is very large, you will not gain anything from compressing it with gzip and then converting it back to a base64 string. For shorter texts the messages will be longer.

 

[Ohanian]

Based on this link the .Net implementation adds 4 bytes (maybe with the length I haven't read it all).

Unless the text is very large, you will not gain anything from compressing it with gzip and then converting it back to a base64 string. For shorter texts the messages will be longer.

the text is a query result and its about 2mb , with compression it becomes 100kb!
i'll try another component (like xceed) to compress the string and decompress it in b4a.

 

[Ohanian]

Hi,

Here's a working solution :

.Net part :

    Function GZipCompress(ByVal s As String) As String
        Dim bytes = Encoding.UTF8.GetBytes(s)
        Using msi = New MemoryStream(bytes)
            Using mso = New MemoryStream()
                Using gs = New GZipStream(mso, CompressionMode.Compress)
                    msi.CopyTo(gs)
                End Using
                Return Convert.ToBase64String(mso.ToArray())
            End Using
        End Using
    End Function

B4A part :

#If Java

import android.Manifest;
import android.content.Context;
import android.os.Build;
import java.io.File;
import java.io.IOException;
import java.util.Collections;
import java.util.List;
import java.util.UUID;
import java.io.ByteArrayOutputStream;
import java.util.zip.GZIPOutputStream;
import java.io.ByteArrayInputStream;
import java.util.zip.GZIPInputStream;

    public byte[] compress(String string) {
        try {
            ByteArrayOutputStream os = new ByteArrayOutputStream(string.length());
            GZIPOutputStream gos = new GZIPOutputStream(os);
            gos.write(string.getBytes());
            gos.close();
            byte[] compressed = os.toByteArray();
            os.close();
            return compressed;
        } catch (IOException ex) {
            return null;
        }
    }

    public String decompress(byte[] compressed) {
        try {
            final int BUFFER_SIZE = 32;
            ByteArrayInputStream is = new ByteArrayInputStream(compressed);
            GZIPInputStream gis = new GZIPInputStream(is, BUFFER_SIZE);
            StringBuilder string = new StringBuilder();
            byte[] data = new byte[BUFFER_SIZE];
            int bytesRead;
            while ((bytesRead = gis.read(data)) != -1) {
                string.append(new String(data, 0, bytesRead));
            }
            gis.close();
            is.close();
            return string.toString();
        } catch (IOException ex) {
            return null;
        }
    }    
   
    // Base64
    public String compressBase64(String strToCompress) {
        byte[] compressed = compress(strToCompress);
        String encoded = android.util.Base64.encodeToString(compressed, android.util.Base64.NO_WRAP);
        return encoded;
    }
   
    public String decompressBase64(String strEncoded) {
        byte[] decoded = android.util.Base64.decode(strEncoded, android.util.Base64.NO_WRAP);
        String decompressed = decompress(decoded);
        return decompressed;
    }

#End If