Multiple files upload challenge

[sangee]

Hi All...

I am making a file upload function in my program which is to upload 100s of text files back to my webserver hosting a PHP script which receives the files and stores them locally.

I am able to upload the files successfully. The challenge and problem I am facing is that the number of files which needs to get uploaded keeps increasing as the file generation is continuous and the upload function is not able to keep up with the speed of file generation. Also the functions starts uploading files which has been already uploaded but waiting for deletion confirmation to be received.

I am using HttpUtils2 for file uploading. Below code does the file upload for me. I call this in a timer which runs every 1 second.

Sub UploadTimer_Tick
Dim fileList As List
Dim n As Int
Dim file1 As String
   fileList = File.ListFiles(File.DirRootExternal & "/Download/ubtxt/Files/")
   fileList.Sort(True)
   lblfilesinque.Text = fileList.Size 
   If fileList.Size > 0 Then
         '   Loop through the list. upload the first file in the queue and exit loop
            For n = 0 To fileList.Size-1
               file1 = fileList.Get (n)
               FileJob = file1
               Dim job2 As HttpJob
               job2.Initialize(FileJob,Me)
               job2.PostFile("http://www.xxxxxxxx.com/uploadimg.php?filename=" & fileList.Get(0),File.DirRootExternal & "/download/ubtxt/Files/", file1)
               Exit ' Exit for loop after processing the first file
            Next                  
   End If
End Sub


Sub JobDone (Job As HttpJob)
Dim Filename As String 
   If Job.Success = True Then
            Select Job.JobName
                  Case "Job1"
                     'Get and set the device status
                     DevStatus(Job.GetString)
                  Case FileJob
                     Filename = Job.GetString
                     Filename = Filename.Trim 
                     File.Delete (File.DirRootExternal & "/Download/ubtxt/Files/",Filename)
                     ToastMessageShow("Upload Ok: " & Filename ,False)
            End Select
   Else
            Log("Error: " & Job.ErrorMessage)
            ToastMessageShow("Error: " & Job.ErrorMessage, True)
   End If
   Job.Release
End Sub

Can anyone please guide and help me in making the function better which will ensure the files already uploaded does not get uploaded again and if possible upload multiple files in one go. Thanks in advance..

Regards,
Sangee

 

[Erel]

Why do you need a timer? Why not send all the files at once? Do not worry about server load. The files will not be sent immediately.

 

[sangee]

Thanks for the response Erel...

As you can notice in my code I am assigning the jobname as the filename itself.

So in the JobDone event the file should get deleted when that particular file is confirmed as uploaded. This is as per the code.

Unfortunately before the webserver can pass on back on the file received confirmation string the upload loop sends the same file again.

Please let me know if this could be a good approach..

1. I wait to see if the file count becomes 10
2. once the file count is 10, i loop through the file list uploading each file with the jobname as the filename
3. The jobdone event will start receiving the file upload confirmation for each individual jobname and delete the respective file.

Only concern would be how to prevent same file from being uploaded again as this could waste bandwidth and time. Will renaming the file extension of the files being uploaded be a good option?

Thanks once again Erel.. i am stuck in this logic of speed of uploads...

Regards,
Sangee

 

[Erel]

You will need to give us more information. Who is creating these files? How large are these files?

My intuition says that you need to use a database instead of all these file...

 

[sangee]

Hi Erel,

These are combination of text and image files. The image files are captured using the internal camera and text files are created with details of the image details. The size of the text and image files are 4KB each.

A part of my program which is a service module already talks to my remote DB to upload many of the details.

I am struggling to get a solution to ensure i upload all the files created continuously to the server with out missing any image or text files.

Thanks,
Sangee

 

[Erel]

You must build a "more organized" solution. There are many ways to implement it. I recommend you to use a simple database.
Whenever you create a new file, you should add this file to the FilesToSend table.

You should have a status column with the values: WAITING and UPLOADING.

- Send all the WAITING files and change their status to UPLOADING. In JobDone if the upload was success you should delete the file and remove it from the table. Otherwise you should set its status to WAITING.

There are many advantages for using a database here. One of them is that if your application crashes you will still be able to upload the files.
Note that when the process starts (FirstTime = True) then you should change the status of all files to WAITING.

 

[sangee]

Erel... thanks for the expert advice.. this proved to be the best method. I followed the method told by you. Now I have all the files being uploaded properly and not repeating. Its amazing to see even 400+ files getting uploaded so easily...

Thanks Erel...

Regards,
Sangee