Android Question FTP Download multiple files problem

[cirollo]

Hi! I'm getting a strange problem when downloading multiple jpg files (1024 files) from a ftp server.
All the files are about 18 Mb, they are very small (200x200) jpg.
the problem is that after 150-170 files the download ends witohut any error.... sometimes at 150, sometimes at 155, and so on, never on the same file number.

I've used the file list event to write a small txt file in which every line is the name of the file to download. the file is correctly create and read (see scaricaimmagini sub)

After the download completed event, I started with the next file.

These are my subs:

Sub BtnRicevi_Click 'ricezione dati
   If FTPMode="L" Then  
     'File Locali
     Dim Answ As Int
     Answ=Msgbox2("Confermi Import Dati?",Main.nomeprog,"Anagrafiche","Annulla","Immagini",LoadBitmap (File.DirAssets, "warning_small.png"))
     If Answ=DialogResponse.POSITIVE Or Answ=DialogResponse.NEGATIVE Then
       If Answ=DialogResponse.POSITIVE Then
         TipoImp="T"
       Else
         TipoImp="I"
       End If
       CircularProgressBar1.Value = 30
       PopolaTabelle
     End If
   End If
   If FTPMode="F" Then
     'ftp
     Dim server As ServerSocket 'Add a reference to the network library
     server.Initialize(0, "")
     If server.GetMyIP = "127.0.0.1" Then  'this is the local host address
          'no connection
       Msgbox2("Nessuna Connessione!",Main.nomeprog,"","Ok","",LoadBitmap (File.DirAssets, "warning_small.png"))
     Else
       Dim Answ As Int
       Answ=Msgbox2("Confermi Import Dati?",Main.nomeprog,"Anagrafiche","Annulla","Immagini",LoadBitmap (File.DirAssets, "warning_small.png"))
       If Answ=DialogResponse.POSITIVE Or Answ=DialogResponse.NEGATIVE Then
         If Answ=DialogResponse.POSITIVE Then
           TipoImp="T"
         Else
           TipoImp="I"
         End If
         TxtLog.Text=" "
         Awake.KeepAlive(True) 'evitiamo lo stand by
         LblStatus.Visible = True
         TxtLog.Visible = True
         BtnInvia.Enabled = False
         BtnRicevi.Enabled = False
         BtnExit.Enabled = False
    
           'download immagini
           'faccio pulizia prima di scaricare i files dall'FTP
           TxtLog.Text = "Eliminazione files obsoleti..." &Chr(10)&TxtLog.Text
           File.Delete(sdRoot,"listaimmagini.txt")
           File.Delete(sdRoot&"/img/","*.jpg")
           TxtLog.Text = "Connessione FTP "& FTPHost&"(" &FTPPort &") in Corso..." &Chr(10)&TxtLog.Text
           CreaFileListaImmagini
         End If
       End If
     End If  
   End If
End Sub

Sub FTP_ListCompleted (ServerPath As String, Success As Boolean, Folders() As FTPEntry, Files() As FTPEntry)
   Log(ServerPath)
   If Success = False Then
     Log(LastException)
   Else
     Dim j As Int
     j=0
     For i = 0 To Folders.Length - 1
       Log(Folders(i).Name)
     Next
     Dim List1 As List
     List1.Initialize
     totimmagini=0
     'ciclo per scaricare i files
     For i = 0 To Files.Length - 1
       Log(Files(i).Name & ", " & Files(i).Size & ", " & DateTime.Date(Files(i).Timestamp))
       Log("/immagini/"&Files(i).Name&"-"&sdRoot&"/img/"&Files(i).Name)
     '   FTP.DownloadFile("/immagini/"&Files(i).Name, False, sdRoot&"/img/", Files(i).Name)
       List1.Add(Files(i).Name)
       File.WriteList(sdRoot,Nfile,List1)
       totimmagini=totimmagini+1
     Next
     TxtLog.Text = "Creato File "&Nfile&Chr(10)&TxtLog.Text
     'scarichiamo il primo file
     numimmagine=1
     Dim Reader As TextReader
     Reader.Initialize(File.OpenInput(sdRoot, "listaimmagini.txt"))
     Dim line As String
     '  Reader.ReadLine
     line = Reader.ReadLine
     FTP.DownloadFile("/immagini/"&line, False, sdRoot&"/img/", line)
   End If
End Sub

Sub FTP_DownloadCompleted (ServerPath As String, Success As Boolean)
  Log(ServerPath & ", Success=" & Success)
   If Success = True Then
  
     'IMMAGINI
       TxtLog.Text = "Download file "&ServerPath &" completato"&Chr(10)&TxtLog.Text
       LblStatus.Text =numimmagine&"/"&totimmagini
       If numimmagine = totimmagini Then
         FTP.Close
         CircularProgressBar1.Value = 100
         Msgbox2("Ricezione completata!",Main.nomeprog,"","Ok","",LoadBitmap (File.DirAssets, "warning_small.png"))
         TxtLog.Visible = False
         LblStatus.Visible = False
         Awake.ReleaseKeepAlive 'togliamo lo standby
         BtnInvia.Enabled = True
         BtnRicevi.Enabled = True
         BtnExit.Enabled = True
         Activity.Finish
         StartActivity(Main)
       Else
         numimmagine = numimmagine +1
         ScaricaImmagini   ' devo scaricare altri files
       End If
  
   Else
     Msgbox2("Errore nella Ricezione!",Main.nomeprog,"","Ok","",LoadBitmap (File.DirAssets, "warning_small.png"))
     FTP.CloseNow
     TxtLog.Visible = False
     Awake.ReleaseKeepAlive 'togliamo lo standby
     BtnInvia.Enabled = True
     BtnRicevi.Enabled = True
     BtnExit.Enabled = True
     Activity.Finish
     StartActivity(Main)
   End If
End Sub

Sub ScaricaImmagini
   Dim Reader As TextReader
   Reader.Initialize(File.OpenInput(sdRoot, "listaimmagini.txt"))
   Dim i As Int
   Dim line As String
   '  Reader.ReadLine
   i=1
   'Msgbox(numimmagine,"")
   Do While i <= numimmagine
     line = Reader.ReadLine
     i=i+1
   Loop
   Log(numimmagine&"-"&line)
   FTP.DownloadFile("/immagini/"&line, False, sdRoot&"/img/", line)
   Reader.Close
End Sub

Any idea? maybe an FTP server issue? (on the ftp server it is installed filezilla server)
regards
ciro

 

[cirollo]

tried to log the download, it simply shows success for the last file downloaded and then nothing else, no error, no warning, nothing.
the files that blocks the process in not the same, sometimes at 170th files, sometimes at 200th, so I think is something time related.

I can make a short video and link it here to show you....

 

[Erel]

1. Try to add Sleep(1000) between each request.
2. Try to Close and then initialize the FTP client after each request.

 

[cirollo]

tried Erel, every 100 files.....

Sub ScaricaImmagini
  Log(ftpjobs.Size)
  For i = 0 To ftpjobs.Size-1
  FTP.DownloadFile("/immagini/"&ftpjobs.Get(i), False, sdRoot&"/img/", ftpjobs.Get(i))
  Wait For FTP_DownloadCompleted (ServerPath As String, Success As Boolean)
  TxtLog.Text = "Download file "&(i+1)&" di "&totimmagini&" "&ServerPath &" completato"&Chr(10)&TxtLog.Text
  If i=100 Or i=200 Or i=300 Or i=400 Or i=500 Or i=600 Or i=700 Or i=800 Or i=900 Or i=1000 Then
  FTP.Close
  Sleep(1000)
  FTP.Initialize("FTP", FTPHost.Trim, 21, FTPUser.Trim, FTPPwd.Trim)
  FTP.PassiveMode=False
  End If
  'i=i+1
'  If Success = False Then
'  Log("Error downloading: " & ServerPath)
'  Log(LastException)
'  End If
  Next
  FTP.Close
End Sub

 

[KMatle]

tried Erel, every 100 files.....

1. What does the server say? (messages/logs). Maybe the server handles your downloads as an attack and blocks you for a while. But even then you receive a success=false
2. I haven't seen any logs of your app (ther must be a message). Please add some additional lines like ("Download started", "Download ended", "Success=True/False", etc.)
3. Where do you store these files. Maybe your Android system is short on storage
4. Did you try the same with B4J? (to prevent memory issues with Android)

 

[cirollo]

1.the server says nothing, it stops download (is filezilla server)
I dont' receive any success=false
2. I put a line like this: log(ftpjobs.Get(i)&"-"&success), always success=true
3. I checked, it has more than 5 gbs free
4. No I haven't tried with b4j, it could be a memory issue?

thx

 

[Erel]

If you like you can send me the project and I'll test it here.

 

[Erel]

I've tested it and found the problem. There were cases where the FTP server stopped to respond. A timeout was not set so the client waited indefinitely.

This issue is fixed in Net v1.75. There is a new TimeoutMs parameter. Its default value is 60000 (60 seconds). You can decrease it if you want it to fail sooner.

 

[cirollo]

I've tested it and found the problem. There were cases where the FTP server stopped to respond. A timeout was not set so the client waited indefinitely.

This issue is fixed in Net v1.75. There is a new TimeoutMs parameter. Its default value is 60000 (60 seconds). You can decrease it if you want it to fail sooner.

thank you Erel!
So, you're saying is a problem on the ftp server, right? (is a cloud server, with filezilla on it, very strange)
If I update the net library, the only thing that I can get is an error message......but no way the server continue to download files...., right?

what about open ftp connection, download file and close....one file at once?

 

[Erel]

Not exactly. There is a problem on the server side however the client can easily handle it with the new timeout setting. Just try to download again.

 

[cirollo]

Not exactly. There is a problem on the server side however the client can easily handle it with the new timeout setting. Just try to download again.

tried now, it stops then start again until the last file. the problem is that from the file that stops the first time until the last one I get alway false and the image file is corrupt.

I think is better to stop the download when it stucks and then start again with the remaining files. How can I do this?

 

[Erel]

Call FTP.CloseNow. Sleep for a few seconds, initialize FTP again and continue downloading.

 

[cirollo]

Call FTP.CloseNow. Sleep for a few seconds, initialize FTP again and continue downloading.

Right! I do that, at the first file with success=false, I raise a messagebox and invite the user to do it again, of course I check if the current file is present on the device avoiding to download it again the second time.....