Android Question Download a lot of files!?

[aidymp]

Hi, I have a lot of files to download, I had these in a zip, downloaded and extracted the zip. this works fine! however these files in my case images (but i dont want to restrict this question to images so they are FILES!) the zip has grown in size and is now around 120meg! as the images change this downloads everytime you run the app! So im looking to download a list of files (I can check sizes, and if they exist prior to doing this thats not the problem!)

I have a list of files stored in a list. (these are just the file names) so I can submit a job and add the url & the filename from the list.

I need the filename to be saved in each case. so a file called file1.jpg must be saved as file1.jpg and so on.

The code I tried was this - but it crashes

For a = 0 To ListOfImages.Size-1
        LogColor(ListOfFiles.Get(a),Colors.Blue)
        Dim Job As HttpJob
        Job.Initialize("file", Me)
        JobsIndex.Put(Job, a)
       Job.Download("http://test.com/apps/"&ListOfFiles.Get(a))
Next

Sub JobDone (Job As HttpJob)
Log("Job "&Job.JobName)
If Job.JobName = "file" Then
    Dim index As Int = JobsIndex.Get(Job) 'find the index of the job
    Log(ListOfFiles.Get(index))
    Dim o As OutputStream
    o = File.OpenOutput(File.DirDefaultExternal,ListOfFiles.Get(index) , False)
    File.Copy2(Job.GetInputStream, o)
    o.Close
    Return
End If
...
end sub

I get an error - immediately like so

httpjob_vvvvvvvvv0 (java line: 138)
java.io.FileNotFoundException: /storage/emulated/0/Android/data/com.aidymp.aidymatic/files/54: open failed: ENOENT (No such file or directory)
at libcore.io.IoBridge.open(IoBridge.java:456)
at java.io.FileInputStream.<init>(FileInputStream.java:76)
at anywheresoftware.b4a.objects.streams.File.OpenInput(File.java:209)
at com.aidymp.aidymatic.httpjob._vvvvvvvvv0(httpjob.java:138)
at com.aidymp.aidymatic.main._jobdone(main.java:4143)
at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:169)
at anywheresoftware.b4a.keywords.Common$5.run(Common.java:996)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:135)
at android.app.ActivityThread.main(ActivityThread.java:5254)
at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:903)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:698)
Caused by: android.system.ErrnoException: open failed: ENOENT (No such file or directory)
at libcore.io.Posix.open(Native Method)
at libcore.io.BlockGuardOs.open(BlockGuardOs.java:186)
at libcore.io.IoBridge.open(IoBridge.java:442)
... 16 more

What could be the problem?

Thanks

Aidy

 

[DonManfred]

use your list
download ONE file
In jobdone your start the download of the next one from your list

Starting thousands httpsjobs will not work

Seethis example... It is using the same technique

 

[aidymp]

use your list
download ONE file
In jobdone your start the download of the next one from your list

Starting thousands httpsjobs will not work

Seethis example... It is using the same technique

Hi, I did have this working at one point but cannot find the code, looking at it again it looks like witchcraft! I have a map with a file name and url, no matter what I do I cant pass the data to this! I keep getting the error where I cannot cast a map to a list?!

Ideally I just want to use my existing map with a filename, and url, or possibly a filename (as i know the url) and a destination folder...

looking at it and you seem to grab something called ("filename") from the list, and the problem is a dont understand how to add something to the list that would have a name!!! as i just have say "filename.jpg","savetothislocation/folder1/temp" in a map?

Ideally the example as a sub is my aim!

If someone made such a sub, I would make a donation

Thanks

Aidy

 

[DonManfred]

See this example. Hope it is of help...

Basically it starts with

  Dim job As HttpJob
  job.Initialize("getfilelist", Me)
  job.Tag = "filelist"
  job.download("http://snapshots.basic4android.de/snapshots.php") ' starts download...
    downloadqueue.Initialize

to get a list of files... see Jobdone Event. Here the files are added to the gobal downloadlist.
One file is downloaded. When read the next one from the "queue" is downloaded...

 

[DonManfred]

Ideally I just want to use my existing map with a filename, and url, or possibly a filename (as i know the url) and a destination folder...

My example above is saving all files to the same path, yes.

But you are able to do more control.

Look this code. Here i use a Map for each download in the Queue.
The url is taken from the map to start the download
the map is set as Tag in the downloadjob.
after the download is finished the Filename and Path to store the file to are taken from the map.

UNTESTED

Sub JobDone(Job As HttpJob)

    Log("JobName: "&Job.JobName&" / Tag = "&Job.Tag&" / Tag = "&Job.Success)
  If Job.Success Then
        If Job.JobName = "getfilelist" Then
            Dim res As String = Job.GetString
            Dim parser As JSONParser
            parser.Initialize(res)
            Dim root As List = parser.NextArray
            For Each colroot As String In root
                Dim m As Map
                m.Initialize
                m.Put("URL","http://snapshots.basic4android.de/"&colroot)
                m.Put("Filename",GetFilename(colroot))
                m.Put("Path",File.DirRootExternal)
                downloadqueue.Add(m)
            Next
            ProgressBar1.Progress = 0
            If downloadqueue.Size > 0 Then
                Dim Job As HttpJob
            Job.Initialize("downloadfile", Me)
                Job.Tag = downloadqueue.Get(0)
                Dim m As Map = downloadqueue.Get(0)
                Log("Download "&m.Get("Filename")&" from "&m.Get("URL"))
                Job.download(m.Get("URL")) ' starts download...
            End If

    else If Job.JobName = "downloadfile" Then
            Log("Save file...")
            filesdownloaded = filesdownloaded +1

            Dim m As Map = Job.Tag ' NEW
            Log("Download of "&m.Get("Filename")&" from "&m.Get("URL")&" finished") ' NEW

            Dim c As Int =  (downloadqueue.Size/100)*filesdownloaded
            ProgressBar1.Progress = Abs(c)

            Dim OutStream As OutputStream
            OutStream = File.OpenOutput(m.Get("Path"), m.Get("Filename"), False) ' Path and Filename are taken frm the Map...
      File.Copy2(Job.GetInputStream,OutStream) ' save the file
      OutStream.Close
      Log(Job.Tag&" written to "&File.DirRootExternal) ' Write the Originalname to the log to see what happens ;-)
         
            If downloadqueue.Size > 0 Then
                For i = downloadqueue.Size-1 To 0 Step -1
                    If downloadqueue.Get(i) = Job.Tag Then
                        downloadqueue.RemoveAt(i)
                    End If
                Next
            End If
         
            If downloadqueue.Size > 0 Then
                Dim Job As HttpJob
            Job.Initialize("downloadfile", Me)
                dim m as map = downloadqueue.Get(0)
                
                Job.Tag = m
            Log("Download "&m.get("URL"))
                Job.download(m.get("URL")) ' starts download...
            End If
         
    End If
  Else
        Log("Error: "&Job.ErrorMessage)
    End If
   
    Job.Release
    For i = 0 To downloadqueue.Size-1
        'Log(downloadqueue.Get(i))
    Next
 
End Sub
Sub GetFilename(fullpath As String) As String
   Return fullpath.SubString(fullpath.LastIndexOf("/") + 1)
End Sub

 

[aidymp]

Thank you! this example is much easier to follow, the previous one seemed to move things from a map to a list and back again, made my head hurt! but was effective!

I have tried this new example and it seems easy to follow, I will take a good look at this later.
this was a major headache for me! as i got someone to do some php, that makes a zip (of images) and I downloaded the zip as I needed the images and new images on the launch of my program, but the zip is getting larger and larger, so my app is getting slower and slower to run!

Thanks

Aidy

 

[DonManfred]

the previous one seemed to move things from a map to a list and back again

Not really.
The previous one was using a global LIST.
I was adding just the url as string to this list.
And this is getting back to generate the download or to store the filename.

The newest one is, honestly, more complex as it is using a Map and not just a string.But the list it the same

And this map is getting back and so on...
But with the addition to transfer more informations to the job.

But you seem to be able to understand it easier

And, if you compare the two versions then ouy´ll see that only a few lines have changed