Bulk file upload - Stuck terribly

[sangee]

Hell All...

I am making a app which needs to transfer 1000s of image files to the web server using a PHP script hosted in the webserver. The app needs to just simply upload the file using http post.

Initially I encountered severe problems in terms of same file being uploaded again and again. Based on a suggestion by Erel I added the facility of using SQlite Db in which the file name row gets marked as "WAITING" or "UPLOADING" and finally gets deleted when the upload is successful.

The above method worked but.. when the internet link goes down due to some chance and the app is not able to reach out to the webserver the files simple gets piled up and the database gets too heavy and starts choking out.

I request if anyone out here has attempted in uploading 1000s of files who can help me please...

:sign0085:


Regards,
Sangee

 

[Erel]

The problem is not in the database. The database can handle millions of records.

Are you continuing to send files when there is no connection?

 

[sangee]

I have ensured to limit the number of upload (threads?) not to exceed 20 as I read in some post. I reduce the thread counter when there is a successful upload or a failure in upload.

An important issue (or some mistake I have made ) I have noticed is that the function which inserts the new file names with the status as "WAITING" does not get listed when another function (upload function) does a.. I have a doubt if the inserts are getting committed or not to the DB. I tried the begin transaction and end transaction method also to no luck. Sometimes the inserts shows up in in the query of upload function after 5 or 6 minutes.One feature of the app is to upload a old file and the latest captured image file alternatively..

Cursor1 = SQL1.ExecQuery2("SELECT * FROM FilesToSend WHERE status = ?", Array As String("WAITING"))

Erel, I am posting the project here.. I request you to kindly go through it and guide me please..

View attachment CamService.zip

 

[Erel]

The 20 threads limit is not relevant anymore. It is handled internally.

You should not initialize the timer in Service_Start. It will be initialized many times and this will cause problems.

About your program logic. I'm sorry but this project is too large for me to go into...

SQL Inserts are committed immediately.

 

[mc73]

Tried sending one file then the next one? Another way I could think of is zipping and sending the zip file.
Just general thoughts, haven't checked your code.