B4J Question Jsch Synchronous FileDownload

[Fr Simon Rundell]

I have a project building an automatic backup downloading tool in B4J but have hit a major snag...

The App contains a B4XTable with information about a download enabling a login to a given server and a path from which it pulls the latest (backup) file and saves it.

To pull the given file, I iterate through the b4xtable and access the server with the right credentials:

Private Sub ftpGo(Server As String, Login As String, Password As String, Path As String)
    
    sftp.Initialize("sftp", Login, Password, Server, 22)
    sftp.SetKnownHostsStore(File.DirData("cmSFTP"), "hosts.txt")
    
    sftp.List(Path)
    
End Sub

This makes the right connection and then I list the files in the specified Path which then raises the _ListCompleted Event

Sub sftp_ListCompleted (ServerPath As String, Success As Boolean, Folders() As SFtpEntry, Files() As SFtpEntry)
    
    If Files.Length> 0 Then
        Log("There are " & Files.Length & " items in this folder")
        Log(Files(Files.Length-1).Name & " date is " & DateTime.Date(Files(Files.Length-1).Timestamp))
        
        If ServerPath.CharAt(ServerPath.Length - 1) <> "/" Then
            Log("Adding final forward slash")
            ServerPath=ServerPath & "/"
        End If
        
        lblFile.Text = "File: " & Files(Files.Length-1).Name
        getFile(ServerPath & Files(Files.Length-1).Name, Files(Files.Length-1).Name)
        
        
    Else
        Log("No Files found in " & ServerPath)
    End If
    
End Sub

Sub getFile(Path As String, Filename As String)
    sftp.DownloadFile(Path, BackupLocation, Filename)
End Sub

Sub sftp_DownloadCompleted (ServerPath As String, Success As Boolean)
    Log(ServerPath & ", Success=" & Success)
    If Success = False Then Log("Download Error: " & LastException.Message)
    sftp.Close
End Sub

The right file is found and passed to geFile so I can download the file.

HOWEVER

This only works when I have a single entry in the B4XTable, otherwise the code just ploughs on through to the next entry in the B4XTable, registering that there are no files in the given folder of the first and second entries on the B4XTable, but it then starts successfully downloading the third.

I assume this is because the sftp.FileDownload action is asynchronous...

Can I / Should I wrap sftp.DownloadFile(Path, BackupLocation, Filename) in a Wait For?

Your advice / experience greatly appreciated.

Simon+

 

[OliverA]

Actually, you have to WAIT FOR several levels AND make your ftpGO method a resumable sub AND wait for it in the loop you use to parse the entries in your B4XTable.

'Make this sub resumable and call with
'Wait For (ftpGo(param1, param2, param3, param4)) complete (success as Boolean)
'Note: Replace param? with appropriate values
Private Sub ftpGo(Server As String, Login As String, Password As String, Path As String) As ResumableSub
    Dim retVal As Boolean 'default to False
    'Let's make sftp local to here
    Dim sftp As SFtp

    Try
        sftp.Initialize("sftp", Login, Password, Server, 22)
        sftp.SetKnownHostsStore(File.DirData("cmSFTP"), "hosts.txt")
        sftp.List(Path)
        'Doing it as per https://www.b4x.com/android/forum/threads/sftp-and-wait-for.107518/post-672207
        'Also as per @Erel, you need to make sure to serialize the calls to the SFtp methods
        Wait For sftp_ListCompleted (ServerPath As String, Success As Boolean, Folders() As SFtpEntry, Files() As SFtpEntry)
        If Files.Length> 0 Then
            'Using some smart string literals
            Log($"There are ${Files.Length} items in this folder"$)
            Log($"${Files(Files.Length-1).Name} date is ${DateTime.Date(Files(Files.Length-1).Timestamp)}"$)

            If ServerPath.CharAt(ServerPath.Length - 1) <> "/" Then
                Log("Adding final forward slash")
                ServerPath=$"${ServerPath}/"$
            End If

            lblFile.Text = $"File: ${Files(Files.Length-1).Name}"$

            'Didn't see BackupLocation in the postes code, so must be a Public variable
            sftp.DownloadFile(ServerPath & Files(Files.Length-1).Name, BackupLocation, Files(Files.Length-1).Name)

            Wait For sftp_DownloadCompleted (ServerPath As String, Success As Boolean)
            Log($"${ServerPath}, Success=${Success}"$)
            If Success = False Then Log($"Download Error: ${LastException.Message}"$)
            retVal = Success
        Else
            Log("No Files found in " & ServerPath)
        End If
    Catch
        Log($"ftpGo Try failure: ${LastException}"$)
    End Try

    'Since SFtp does not have an IsInitialized, wrap Close method in a try/catch block
    Try
        sftp.Close
    Catch
        Log(LastException)
    End Try
    Return retVal
End Sub

Note: Code untested, may be buggy

 

[Erel]

Explanation here: [B4X] Resumable subs that return values (ResumableSub)

 

[Fr Simon Rundell]

Thank you. I found my way to Resumable Sub a little bit before.
I now have it working and this is what I discovered...

1. Having got the result of the contents of a folder with sftp_ListCompleted, the connection appears to have been closed.
2. It was therefore necessary to reconnect and reinitialize the stfp connection. Without this re-connection, it generated a RTE saying sftp needed to be initialized first.

This is the code of this function now. The data is extracted by working through the B4XTable which calls this function for a given server entry:

Private Sub ftpGo(Name As String, Server As String, Login As String, Password As String, Path As String) As ResumableSub
   
    Dim retVal As Boolean = False
    Dim sftp As SFtp


    Try
        sftp.Initialize("sftp", Login, Password, Server, 22)
        sftp.SetKnownHostsStore(File.DirData("cmSFTP"), "hosts.txt")
        sftp.List(Path)

        Wait For sftp_ListCompleted (ServerPath As String, Success As Boolean, Folders() As SFtpEntry, Files() As SFtpEntry)
        If Files.Length> 0 Then
            'Using some smart string literals
            Log($"There are ${Files.Length} items in the backup folder of ${Name}"$)
            Log($"Found: ${Files(Files.Length-1).Name} date is ${DateTime.Date(Files(Files.Length-1).Timestamp)}"$)

            If ServerPath.CharAt(ServerPath.Length - 1) <> "/" Then
                Log("Adding final forward slash")
                ServerPath=$"${ServerPath}/"$
            End If

            ' Visual Feedback of downloading
            lblFile.Text = $"File: ${Files(Files.Length-1).Name}"$
            lblServer.Text = $"Server: ${Server}"$
            lblPath.Text = $"Path: ${Path}"$

            ' it looks like sftp is closed after sftp_ListCompleted so reconnect
            sftp.Initialize("sftp", Login, Password, Server, 22)
            sftp.SetKnownHostsStore(File.DirData("cmSFTP"), "hosts.txt")
           
            sftp.DownloadFile(ServerPath & Files(Files.Length-1).Name, BackupLocation, Files(Files.Length-1).Name)
           
            Wait For sftp_DownloadCompleted (ServerPath As String, Success As Boolean)
                imgLoading.Hide
                Log($"${Name} has been backed up"$)
                toastMessage.Show($"${Name} has been backed up"$)
                If Success = False Then Log($"Download Error: ${LastException.Message}"$)
                retVal = Success
                sftp.Close
           
' Move onto next B4XTable Entry (easier to track than a ForEach loop)
            If (currentEntry+1)<=tblBackuplist.Size Then
                currentEntry=currentEntry+1
                backupSingleEntry(currentEntry)
            Else
                toastMessage.Show("All sites downloaded")
                'Now reset all labels ready for next run
                imgLoading.Hide
                lblServer.Text="Server:"
                lblPath.Text="Path:"
                lblFile.Text="File:"
                lblProgressBar="Ready"
                prgTransfer.Progress=0
                sftp.CloseNow              
            End If
        Else
            Log($"No Files found in ${ServerPath}"$)
            sftp.Close
        End If
    Catch
        Log($"ftpGo Try failure: ${LastException}"$)
        sftp.Close
    End Try

    'Since SFtp does not have an IsInitialized, wrap Close method in a try/catch block
    Try
        sftp.Close
    Catch
        Log($"Failed to close sftp with error ${LastException}"$)
    End Try
    Return retVal
End Sub

Now, this is interesting: in the sfpt_DownloadProgress event, the amount of TotalDownloaded sometimes shows a negative number for the last half of the download (from about 48% onward). I don't know if this is a misleading figure from the library, from the FTP server or an error in my conversion to screen, but during a download it will sometimes count up to 48& and then downwards to 0% and then successfully download. Am I being stupid? Converting to TotalDownloaded value with Abs() doesn't seem to help at all.

Sub sftp_DownloadProgress (ServerPath As String, TotalDownloaded As Long, Total As Long)
    Dim progressMeasure As String
    progressMeasure = "Downloaded "
    progressMeasure = $"Downloaded ${Round(TotalDownloaded / 1024)}KB"$
    If Total > 0 Then progressMeasure = progressMeasure & " out of " & Round(Total / 1024) & "KB (" & Round2(((TotalDownloaded / Total) * 100), 2) & "%)"
    lblProgressBar.Text = progressMeasure
    ' different visual for download - the progress bar
       prgTransfer.Progress =(TotalDownloaded / Total) *100
End Sub

It works functionally now, but I can't ship it until I resolve this TotalDownloaded problem.