Android Question FTP Download multiple files problem

[cirollo]

Hi! I'm getting a strange problem when downloading multiple jpg files (1024 files) from a ftp server.
All the files are about 18 Mb, they are very small (200x200) jpg.
the problem is that after 150-170 files the download ends witohut any error.... sometimes at 150, sometimes at 155, and so on, never on the same file number.

I've used the file list event to write a small txt file in which every line is the name of the file to download. the file is correctly create and read (see scaricaimmagini sub)

After the download completed event, I started with the next file.

These are my subs:

Sub BtnRicevi_Click 'ricezione dati
   If FTPMode="L" Then  
     'File Locali
     Dim Answ As Int
     Answ=Msgbox2("Confermi Import Dati?",Main.nomeprog,"Anagrafiche","Annulla","Immagini",LoadBitmap (File.DirAssets, "warning_small.png"))
     If Answ=DialogResponse.POSITIVE Or Answ=DialogResponse.NEGATIVE Then
       If Answ=DialogResponse.POSITIVE Then
         TipoImp="T"
       Else
         TipoImp="I"
       End If
       CircularProgressBar1.Value = 30
       PopolaTabelle
     End If
   End If
   If FTPMode="F" Then
     'ftp
     Dim server As ServerSocket 'Add a reference to the network library
     server.Initialize(0, "")
     If server.GetMyIP = "127.0.0.1" Then  'this is the local host address
          'no connection
       Msgbox2("Nessuna Connessione!",Main.nomeprog,"","Ok","",LoadBitmap (File.DirAssets, "warning_small.png"))
     Else
       Dim Answ As Int
       Answ=Msgbox2("Confermi Import Dati?",Main.nomeprog,"Anagrafiche","Annulla","Immagini",LoadBitmap (File.DirAssets, "warning_small.png"))
       If Answ=DialogResponse.POSITIVE Or Answ=DialogResponse.NEGATIVE Then
         If Answ=DialogResponse.POSITIVE Then
           TipoImp="T"
         Else
           TipoImp="I"
         End If
         TxtLog.Text=" "
         Awake.KeepAlive(True) 'evitiamo lo stand by
         LblStatus.Visible = True
         TxtLog.Visible = True
         BtnInvia.Enabled = False
         BtnRicevi.Enabled = False
         BtnExit.Enabled = False
    
           'download immagini
           'faccio pulizia prima di scaricare i files dall'FTP
           TxtLog.Text = "Eliminazione files obsoleti..." &Chr(10)&TxtLog.Text
           File.Delete(sdRoot,"listaimmagini.txt")
           File.Delete(sdRoot&"/img/","*.jpg")
           TxtLog.Text = "Connessione FTP "& FTPHost&"(" &FTPPort &") in Corso..." &Chr(10)&TxtLog.Text
           CreaFileListaImmagini
         End If
       End If
     End If  
   End If
End Sub

Sub FTP_ListCompleted (ServerPath As String, Success As Boolean, Folders() As FTPEntry, Files() As FTPEntry)
   Log(ServerPath)
   If Success = False Then
     Log(LastException)
   Else
     Dim j As Int
     j=0
     For i = 0 To Folders.Length - 1
       Log(Folders(i).Name)
     Next
     Dim List1 As List
     List1.Initialize
     totimmagini=0
     'ciclo per scaricare i files
     For i = 0 To Files.Length - 1
       Log(Files(i).Name & ", " & Files(i).Size & ", " & DateTime.Date(Files(i).Timestamp))
       Log("/immagini/"&Files(i).Name&"-"&sdRoot&"/img/"&Files(i).Name)
     '   FTP.DownloadFile("/immagini/"&Files(i).Name, False, sdRoot&"/img/", Files(i).Name)
       List1.Add(Files(i).Name)
       File.WriteList(sdRoot,Nfile,List1)
       totimmagini=totimmagini+1
     Next
     TxtLog.Text = "Creato File "&Nfile&Chr(10)&TxtLog.Text
     'scarichiamo il primo file
     numimmagine=1
     Dim Reader As TextReader
     Reader.Initialize(File.OpenInput(sdRoot, "listaimmagini.txt"))
     Dim line As String
     '  Reader.ReadLine
     line = Reader.ReadLine
     FTP.DownloadFile("/immagini/"&line, False, sdRoot&"/img/", line)
   End If
End Sub

Sub FTP_DownloadCompleted (ServerPath As String, Success As Boolean)
  Log(ServerPath & ", Success=" & Success)
   If Success = True Then
  
     'IMMAGINI
       TxtLog.Text = "Download file "&ServerPath &" completato"&Chr(10)&TxtLog.Text
       LblStatus.Text =numimmagine&"/"&totimmagini
       If numimmagine = totimmagini Then
         FTP.Close
         CircularProgressBar1.Value = 100
         Msgbox2("Ricezione completata!",Main.nomeprog,"","Ok","",LoadBitmap (File.DirAssets, "warning_small.png"))
         TxtLog.Visible = False
         LblStatus.Visible = False
         Awake.ReleaseKeepAlive 'togliamo lo standby
         BtnInvia.Enabled = True
         BtnRicevi.Enabled = True
         BtnExit.Enabled = True
         Activity.Finish
         StartActivity(Main)
       Else
         numimmagine = numimmagine +1
         ScaricaImmagini   ' devo scaricare altri files
       End If
  
   Else
     Msgbox2("Errore nella Ricezione!",Main.nomeprog,"","Ok","",LoadBitmap (File.DirAssets, "warning_small.png"))
     FTP.CloseNow
     TxtLog.Visible = False
     Awake.ReleaseKeepAlive 'togliamo lo standby
     BtnInvia.Enabled = True
     BtnRicevi.Enabled = True
     BtnExit.Enabled = True
     Activity.Finish
     StartActivity(Main)
   End If
End Sub

Sub ScaricaImmagini
   Dim Reader As TextReader
   Reader.Initialize(File.OpenInput(sdRoot, "listaimmagini.txt"))
   Dim i As Int
   Dim line As String
   '  Reader.ReadLine
   i=1
   'Msgbox(numimmagine,"")
   Do While i <= numimmagine
     line = Reader.ReadLine
     i=i+1
   Loop
   Log(numimmagine&"-"&line)
   FTP.DownloadFile("/immagini/"&line, False, sdRoot&"/img/", line)
   Reader.Close
End Sub

Any idea? maybe an FTP server issue? (on the ftp server it is installed filezilla server)
regards
ciro

 

[edgar_ortiz]

Hi Cirollo,

What is the problem?

Can you have a log for every file downloaded?

Regards,

Edgar

 

[sorex]

what is the point of the loop with the line fetch when you only call the ftp outside the loop?
it will only download the last line/link of the text file.

is such cases I put the downloads in a map or list and process them one by one.

if succeeded remove it from the map.
if not try again or try the next one if there is one.

 

[cirollo]

@sorex
yes you're right, I put on a txt file the names of the file that must be downloaded
ad example:

file1.jpg
file2.jpg
file3.jpg
file4.jpg
..........
file1024.jpg

after downloading file1.jpg I jump to the second line of the txt file using reader.readline.
I agree a map is better but I think the problem is not related to txt file. The files are downloaded in the correct order....

@edgar_ortiz
I've a log istruction, on the log panel it stops too, never come to the last file......

could be a timeout problem?

ciro

 

[OliverA]

For i = 0 To Files.Length - 1
       Log(Files(i).Name & ", " & Files(i).Size & ", " & DateTime.Date(Files(i).Timestamp))
       Log("/immagini/"&Files(i).Name&"-"&sdRoot&"/img/"&Files(i).Name)
     '   FTP.DownloadFile("/immagini/"&Files(i).Name, False, sdRoot&"/img/", Files(i).Name)
       List1.Add(Files(i).Name)
       File.WriteList(sdRoot,Nfile,List1)
       totimmagini=totimmagini+1
     Next

I'm guessing the above code originally did the downloading of the files (the FTP.DownloadFile was not commented out)? FTP.DownloadFile does not block, so essentially your creating one FTP download after the other before the previous one finishes. Depending on download size, this could be hundreds of requests to download. This may be the issue. You may need to implement some sort of "Active" flag and restrict yourself to one download at a time, or is WAIT FOR.

 

[cirollo]

FTP.DownloadFile does not block, so essentially your creating one FTP download after the other before the previous one finishes. Depending on download size, this could be hundreds of requests to download

that's why is commented, I start the first download, then at every ftpdownloadcompleted event goes to the next one, in this way:

ftp_listcompleted >> first file to download >> ftp_downloadcompleted >> scaricaimmagini (reads from txt file the next file) then ftp_download new file....

when it wasn't commented I supposed was that the issue, so I make a new loop outside the ftplist event.

 

[OliverA]

I finally get it (didn't see it at first - how you were continuing the download). Is this in a different activity? If so, log something in Activity_Pause and see if Android is killing off your Activity.

 

[cirollo]

no, its another sub on the same code activity. android doesn't kill my process, it simply stops without any issue....

 

[OliverA]

If you post a minimal app that can reproduce this issue, that would probably help in trying to figure out what is going on.

 

[cirollo]

Still having this problem, tried also to use a list instead of a csv file, sometimes it stops for a while then goes on, sometimes it stucks!

 

[Erel]

1. Create a new project.
2. Add this code and run it:

Sub DownloadAll
   Dim ftp As FTP
   ftp.Initialize("ftp", ...)
   For Each f As String In File.ReadList(...)
     ftp.DownloadFile("/serverpath/" & f, False, File.DirInternal, f)
     Wait For ftp_DownloadCompleted (ServerPath As String, Success As Boolean)
     If Success = False Then
       Log("Error downloading: " & ServerPath)
       Log(LastException)
     End If
   Next
   ftp.Close
End Sub

 

[cirollo]

1. Create a new project.
2. Add this code and run it:

Sub DownloadAll
   Dim ftp As FTP
   ftp.Initialize("ftp", ...)
   For Each f As String In File.ReadList(...)
     ftp.DownloadFile("/serverpath/" & f, False, File.DirInternal, f)
     Wait For ftp_DownloadCompleted (ServerPath As String, Success As Boolean)
     If Success = False Then
       Log("Error downloading: " & ServerPath)
       Log(LastException)
     End If
   Next
   ftp.Close
End Sub

sorry for the delay, I was out of office in the last 2 weeks....by the way also this sub stops at roughly 200th-220th download.....

 

[Knoppi]

Try to run Erels code in a service, may be a timeout problem.
If there is any Error Message ?

I have nearly the same code in my app and it works.

 

[cirollo]

no error at all, it simply stucks!

the total number of jpg file that I must download is 1029, each jpg is only 40-50 kb.....

 

[cirollo]

no error at all, it simply stucks!

the total number of jpg file that I must download is 1029, each jpg is only 40-50 kb.....

tried also in this way but it stops again!!

Sub ScaricaImmagini
   Log(ftpjobs.Size)
   For i = 0 To ftpjobs.Size-1
     FTP.DownloadFile("/immagini/"&ftpjobs.Get(i), False, sdRoot&"/img/", ftpjobs.Get(i))
     Wait For FTP_DownloadCompleted (ServerPath As String, Success As Boolean)
     TxtLog.Text = "Download file "&(i+1)&" di "&totimmagini&" "&ServerPath &" completato"&Chr(10)&TxtLog.Text
     If i=100 Or i=200 Or i=300 Or i=400 Or i=500 Or i=600 Or i=700 Or i=800 Or i=900 Or i=1000 Then
       FTP.Close
       Sleep(1000)
       FTP.Initialize("FTP", FTPHost.Trim, 21, FTPUser.Trim, FTPPwd.Trim)
       FTP.PassiveMode=False
     End If
     'i=i+1
'     If Success = False Then
'       Log("Error downloading: " & ServerPath)
'       Log(LastException)
'     End If
   Next
   FTP.Close
End Sub

 

[Erel]

If i=100 Or i=200 Or i=300 Or i=400 Or i=500 Or i=600 Or i=700 Or i=800 Or i=900 Or i=1000 Then

If i Mod 100 = 0 And i > 0 Then

Why aren't you checking Success and LastException?

 

[cirollo]

If i Mod 100 = 0 And i > 0 Then

Why aren't you checking Success and LastException?

yes Erel, you're right, it was just for a quick test......

 

[Erel]

What was the result? Are there any errors?

 

[cirollo]

no error at all, it simply stucks!

the total number of jpg file that I must download is 1029, each jpg is only 40-50 kb.....

see my older post, tried also to close ftp connection and reopen it every 100 files but still stops.....around 200th file.....

 

[Knoppi]

With FTP can many problems occur eg authorizations, not created directories , damaged files, etc., but in all these cases you get a error message.
log your download maybe you have a problem with a file
without a error message we can not really help