Android Question Trying to download a test.gz file

Midimaster · Jul 2, 2020

Hi, I'm testing around with file tranfer from my server to the file system of my app. I use OKHttpUtil2 and everything looks like working fine:

The file "test.gz" on the server is a 33MByte gzip-compressed file. It contains a single 80MB data-file
The file "test.abc" is a exact copy of the "test.gz", but I changed the extension to "abc"

When I start the dowload for both files, the time for loading "test.gz" is the same as for "test.abc". OK!

But the file size is different at the end!!!

The "test.abc" has still the 33MByte as expected.

The "test.gz" has 80MByte and this is exactly the size the file inside the gz has. It looks like the file was unzipped automatically.

I do not use any de-compressing code!

B4X:

Sub Globals
    Private Button1 As Button
    Private MyJob As  HttpJob
    Private Zeit As Long
End Sub

Sub Activity_Create(FirstTime As Boolean)
...   
      MyJob.Initialize("MyJob", Me)
End Sub

Sub Button1_Click
    Zeit=DateTime.Now
    MyJob.Tag="test.abc"
    'MyJob.Tag="test.gz"
    MyJob.Download("http://www.MyServer.de/temp/" & MyJob.Tag)
    MyJob.JobName="check"
End Sub


Sub Jobdone(Job As HttpJob)
        Log("Jobname=" & Job.JobName)
        If Job.Success=True Then
                Log("Input Stream from internet")
                    Private MyStream As OutputStream
                    MyStream=File.OpenOutput(File.DirInternal,Job.Tag,False)
                    File.Copy2(Job.GetInputStream(), MyStream)
                    MyStream.Close
                Log("Zeit download=" & (-Zeit+DateTime.Now))
                    Dim Liste As List
                    Liste=File.ListFiles(File.DirInternal)
                    For i=0 To Liste.Size-1
                Log("Files=" & Liste.Get(i) & " " & File.Size(File.DirInternal,Liste.Get(i)))
                        Next
         Else   
            Log("Error beim Download " & Job.ErrorMessage)
        End If
        Job.Release
End Sub

Is this normal?

Erel · Jul 3, 2020

You should never have a JobDone sub.
[B4X] OkHttpUtils2 with Wait For

OkHttpUtils2 supports gzipped content. Many servers return such content and it is decompressed automatically. The exact behavior depend on the request and response headers.

Sandman · Jul 3, 2020

(Regarding the JobDone, it's #10 in the thread about code smells.)

Midimaster · Jul 3, 2020

Ok, I will change the callback way.

But I do not understand, why OkHTTPUtil2 does the job without a need. Can I suppress it? What, when I want to receive the file still compressed?

And why not doing the job completely?
After "auto-compressing" the OkHTTPUtil2 did not change the extension? It loads a compressed file (with correct extension ".gz" ) and then saves the de-compressed file but still with the ".gz"-extension? Is this a bug? My app cannot know, that this changed to a not-compressed file, when the extension still says "yes it still is"! Opening it with a de-compressing algo would cause problems, or?

DonManfred · Jul 3, 2020

Midimaster said:
What, when I want to receive the file still compressed?

Do not use .gz Files. Use zip, rar. Or reconfigure your httpserver not to send compressed streams.

Midimaster said:
After "auto-compressing" the OkHTTPUtil2 did not change the extension?

No. There is no filename or extension. It is just a stream of bytes returning from your server. YOU do set the filename when you save the data.

Midimaster · Jul 3, 2020

It's not the server, which builds the ".gz"-file. It's me. I already uploaded a compressed file to the server. So I'm still thinking whether I should change to ZIP for this project or use it as a "given feature"... It depents on whether I can expect a speed advantage on loading the file in the app and then de-compressing it, or simply open a already de-compressed file.

"YOU do set...."
Oh yes!!!! I'm so stupid!
thanks for answer

Midimaster · Jul 3, 2020

Ok, I changed to Wait For and understand it. At the end the sub continues without stopping the smartphone.

Because the Filesize is 33MByte and the transfer needs some time to be finished, I would now like to try a new way.
Is there a possibility to get informed about the number of the transfered byte?
I would like to use async loading files from internet and from filesystem.
But I'm a beginner and all these tutorials use a lot of new stuff and additional libraries. Often the tutorials are out-of-date and link to a link to a.... Too many "wrong" informations for a beginner, who only wants to code a small learning app to understand.

So now I start a new threat after searching for a hour. please help me with an easy to understand solution

What is the most basic way to load a file async?

I would like to use the first 10000 Byte already, when the transfer is still running. Is there a way?
I would like to know how many bytes are already transfered for a progress bar.

I think i got a first success with in-streaming a file from the filesystem:

B4X:

Sub Globals
    Private Button1 As Button
    Private Zeit As Long
    Private Summe As Int
End Sub

Sub Activity_Create(FirstTime As Boolean)
    Activity.LoadLayout("Main.bal")
End Sub

Sub Button1_Click
    Zeit=DateTime.Now
    NeuesLaden("Oma.jpg")
End Sub

Sub NeuesLaden(FileNamen As String)
    Dim locStream As AsyncStreams
    locStream.Initialize(File.OpenInput(File.DirInternal,FileNamen),Null,"Laden")
End Sub

Sub Laden_NewData(Buffer() As Byte)
    Summe=Summe + Buffer.Length
    Log("new data coming" & (DateTime.Now-Zeit ) & " Lang=" & Summe)
End Sub

Sub Laden_Terminated
    Log("Ende des Streams" & (DateTime.Now-Zeit ))
End Sub

Now I try to do the same with files coming from the internet...

drgottjr · Jul 3, 2020

not sure if you resolved your .gz problem or if you simply moved on without any resolution. i have some comments:
1) the download could take place as a service. this leaves the main portion of the app to continue. if that served some purpose.
2) okhttputil2 sends minimal headers, and - as has been mentioned - supports compressed files. an app of mine downloads updates in a .zip archive via okhttputils2. it arrives compressed.
3) i put a text.gz type of file on my server and on google cloud. i downloaded each via okhttputils2. they both arrived compressed.
4) i changed the file name from test.gz to test.abc and downloaded. they maintained the original size of the compressed file.
5) okhttputils2 is not uncompressing the file. neither my server not google cloud are uncompressing the file when downloaded using okhttputils2. that leaves your server. if you'd provide a valid url and leave test.gz and test.abc out there for pickup, i'd be happy to see how it goes.
6) as was also mentioned previously, servers can be instructed to compress/uncompress files when transferring. you can set the appropriate request header when performing your httpjob.download. you'll have to read the necessary documentation. it's not immediately clear which one or ones you need to add to the request. you might have to experiment a little. without being able to test with your files on your server, it's difficult to know for sure what's happening.

Midimaster · Jul 3, 2020

I moved on... I resolved the problem with my ".abc" workaround. The post here was only for understanding why it happened.

to 1.
i think i will need this one day. At the moment i'm trying to find harmless beginner solutions, where i can understand what happens.
to 2.
also in my tests trad. ZIPs arrives compressed.
to 3.
this is interesting. At my smartphone the transfer time was as short as expected for 33MB (=zipped) but the file size was 80MB at the end.
to 4.
this was my workaround idea. I named the zipped file "*.abc" on the server and it arrived compressed and saved size was also 33MB. Perfect.
to 5.
Oh thank you. Think about it... the data inside are not for public.. perhaps i can fake a file. I will send you a p.m. next days.
to 6.
i'm not interested that the server is doing any job with the file. no no, the file was already compressed, when I put it on the server.

thank you for your comments

drgottjr · Jul 3, 2020

my understanding was the .gz file was arriving uncompressed (and when saved - uncompressed - still had the .gz suffix). my test with .gz files showed (me) that they arrived compressed (.zip or .gz files). that a file - any file - is saved with a .gz suffix, is controlled by the programmer. she can assign any suffix desired to any file. if you saved an uncompressed file with a .gz suffix, the uncompressed file will have .gz as its suffix.

as to my #6, i meant to say that if, in fact, the server was uncompressing .gz files unbeknownst to you, you can send a request header not to do that (you could ask the server's administrator). again, without a test using your server and your files, i can't know whether or not your server needs to be told not to uncompress .gz files. on the other hand, if that is what the server is doing, at least you know not to save it with a .gz suffix or to "uncompress" it again. (or you could name the file with a .zip suffix, since the server apparently handles those correctly.)

regardless of how you download a file, it's useful to know whether your sever is configured to handle .gz files differently from .zip files. you'll run into the same issue whether using httputils2 or async.

Android Question Trying to download a test.gz file

Midimaster

Active Member

Erel

B4X founder

Sandman

Expert

Midimaster

Active Member

DonManfred

Expert

Midimaster

Active Member

Midimaster

Active Member

drgottjr

Expert

Midimaster

Active Member

drgottjr

Expert

Similar Threads

Android Question Trying to download a test.gz file

Active Member

B4X founder

Expert

Active Member

Expert

Active Member

Active Member

Expert

Active Member

Expert

Similar Threads

Privacy & Transparency

Privacy & Transparency