-
Welcome to B4X forum!
B4X is a set of simple and powerful cross platform RAD tools:
- B4A (free) - Android development
- B4J (free) - Desktop and Server development
- B4i - iOS development
- B4R (free) - Arduino, ESP8266 and ESP32 development
-
You are all welcome to join our new Facebook page: https://www.facebook.com/groups/b4x.developers
Android Question Multi core processing in Android
- Thread starter incendio
- Start date
- Similar Threads Similar Threads
How to ask OS to create threads? With threading library?
I am still new in sqlite features, and when using multi threads, I divided data into available threads, for example, there are 100 records & 4 threads, I assigned each threads with 25 records.
So far, performance is better, with 4 threads, at least 6-7 times faster.
The bottle neck is when downloading data with RDC, no matter how many thread created, finished time still the same with 1 thread.
Is this a characteristics of httputtils or connection pooling by jetty?
Last edited:
Upvote
0
Inserting 100 records to a database should be a very quick operation if done correctly.
Can you post the code?
As I wrote above SQLite support for multithreading is very delicate. You can easily corrupt your database.
HttpUtils2 sends each request on a different thread. You cannot make it faster by adding more threads.
Can you post the code?
As I wrote above SQLite support for multithreading is very delicate. You can easily corrupt your database.
HttpUtils2 sends each request on a different thread. You cannot make it faster by adding more threads.
Upvote
0
Do you really need to retrieve 20000 records each time? Or just the first time? What's the volume of data? And what's the accepted duration to write these records to the dabase? Because, enclosed in a transaction, that should be done in less than 30 seconds.
Upvote
0
First time, data from server (+/-20000 records) uploaded to devices. By the time goes by, could be daily/weekly/monthly, data from server gets update,add, and delete(rarely). This new data must upload again to devices.
I have to alternatives to upload this data
1) dirty but simpler, delete all data from devices and replaced it with new data from server (which is currently I choose)
2) complex but efficient, upload only new and modified data from server to devices
Alternative 2 is complex cause must keep a log on server about new data, inserted data & deleted data and then record every device id and keep tracks which device id has updated its data and which one has not. Also must consider device that is not updated its data, but server already update/delete the data. If running well, device could be more than 250.
Volume of data, for first time is about 20000 rows x 8 columns. Column data type, 3 int, 1 char(3), 2 varchar(60) & 1 varchar(4).
I always do sql operations within transaction. With current data, single thread, time to write records is about 18 secs. If speed is linear, than when data reach above 30000, it will took more than 30 secs. But speed is also depend on how fast internal SD Card write & read and how fast cpu on device.
With multi thread, performance increased, see the attached diagram, but as Erel said, that could be corrupted database, I began to consider alternative 2.
Last edited:
Upvote
0
Atleast it seems your downloads are not being threaded for some reason. Without seeing code it is impossible to say, also I am not familiar with RDC.
However, since all HttpUtils2 will be raising the same JobDone back on to the main thread, you need to ensure that JobDone delegates the tasks off the Main thread, and the main thread remains empty.
However, since all HttpUtils2 will be raising the same JobDone back on to the main thread, you need to ensure that JobDone delegates the tasks off the Main thread, and the main thread remains empty.
Upvote
0
You were right, downloads is was not threaded, in runs in sequence.
Here are my codes to create 4 request to RDC
B4X:
Sub GetData
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data1"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data1")
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data2"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data2")
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data3"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data3")
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data4"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data4")
End Sub
Sub JobDone(Job As HttpJob)
If Job.Success = true Then
If Job.JobName = "DBRequest" Then
Dim result As DBResult = reqM.HandleJob(Job)
If result.Tag = "get_data1" OR & _
result.Tag = "get_data2" OR & _
result.Tag = "get_data3" OR & _
result.Tag = "get_data4" Then
Log(DateTime.Time(DateTime.now))
UpdateData(result)
End If
End If
Job.Release
End Sub
Sub UpdateData(result As DBResult)
Try
Private ValC1,ValC2,ValC3,ValC4 As String
Private ValI1,Val2,Val3 As Int
For Each records() As Object In result.Rows
ValI1 = records(6)
ValI2 = records(0)
ValC1 = records(1)
ValC2 = records(2)
ValC3 = records(3)
ValC4 = records(4)
ValI3 = records(5)
SQL.AddNonQueryToBatch("insert into table(Field1,
Field2,Field3,Field4,Field5,Field6,Field7) values(?,?,?,?,?,?,?)",Array As Object(ValI1,ValI2,ValC1,ValC2,ValC3,ValC4,ValI3))
Next
SQL.ExecNonQueryBatch("InsData")
Catch
Log(LastException)
End Try
End Sub
Sub InsData_NonQueryComplete(Success As Boolean)
Log(DateTime.Time(DateTime.now))
End SubAlso, there is no transaction when using SQL.ExecNonQueryBatch cause I read, it is already handle internally. When using 1 thread and SQL.ExecQuery, I used it within transaction.
Last edited:
Upvote
0
No, they are NOT in order
Note:
and httpjobs runs parallel and in fact your code
B4X:
Sub GetData
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data1"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data1")
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data2"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data2")
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data3"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data3")
Dim cmd As DBCommand
cmd.Initialize
cmd.Name = "get_data4"
DBRequestManager.ExecuteQuery(cmd, 0, "get_data4")
End SubStarts 4 jobs and run them all in background.
for ex. get_data4 could return first.
I suppose that you need to take care of the order if that is important to you
Upvote
0
You were right about the order, some time the last one could finished first.
But I think, it was not runs in parallel.
I got something like these from the log
10:00:30
handling job xxxx
10:00:33
handling job xxxx
10:00:37
handling job xxxx
10:00:40
handling job xxxx
From the log, each seem that every job executed after previous job finished.
But I think, it was not runs in parallel.
I got something like these from the log
10:00:30
handling job xxxx
10:00:33
handling job xxxx
10:00:37
handling job xxxx
10:00:40
handling job xxxx
From the log, each seem that every job executed after previous job finished.
Upvote
0
I dont know what the requestmanager does. Maybe he take care of the order. But maybe it is just accidental that the order is correct.
I DONT know and not worked with dbrequestmanager before. But if it just fires 4 httpjobs like i would expect (after read the sidenotes about httpjob in the docu) then the order is not garanteed.
Maybe we need @Erel ´s wisdom to answer this
Upvote
0
Try to create a get_data5 which (server side) just return a special id or something. Not a list of rows. execute a fast query there
So the get_data5 should be answer faster than the others and the result should be return faster in JobDone
So the get_data5 should be answer faster than the others and the result should be return faster in JobDone
Upvote
0
You misunderstood, may be my lack of english made me not explain clearly, sorry.
You were right about order, it is not guaranteed that Job #3 will finished before Job # 4. Sometime Job order #4 could be finished before Job order #3.
What I mean here is, jobs were not executed in parallel. If it runs in parallel, I would expect interval to execute between jobs will be the same or not much different. May be the log would be something like these :
10:00:30
Handling job #3
10:00:30
Handling job #2
10:00:31
Handling job #1
10:00:32
Handling job #4
Upvote
0
Maybe the database server handles the requests synchronously.
You can see performance numbers of a server app created with B4J and MySQL: http://www.b4x.com/android/forum/threads/37502/#content
It is similar to RDC.
You can see performance numbers of a server app created with B4J and MySQL: http://www.b4x.com/android/forum/threads/37502/#content
It is similar to RDC.
Upvote
0
1,4 Mb for the whole database is not too heavy to consider a complete download each time, but if your user has a very slow connection or a limited volume of data per month, he/she won't be happy to have to download everything each time he/she uses your app.
In your scenario (multiple readers for a remote database), the option 2 is the way to go and is not as complex as you think:
1) Add a timestamp to each record and query from the client only the records that are newer than the latest timestamp retrieved with the last download.
2) Create a new table on the server storing the ID of all deleted records with the timestamp of their deletion (this table could be filled automatically with a trigger). Do the same kind of query as above to know the records to delete.
Upvote
0
This database server is also used in LAN and I am pretty sure that it is set for asynchronously, but I will check again its JDBC.
Is there any settings in I should also check in file c3p0.properties (file from jetty) to make sure that it handle request in asynchronously mode?
Upvote
0
Thanks for your suggestion. Considering that there is a potential corrupted sqlite database in devices, which is a more big problems, it seem that this the only solution.
I will modified this idea a bit :
1) create new table on server & client, consist only 1 field,type incremental int id
2) every time records on server get update/insert/delete, id on server's table increased by 1
3) when users login to app, app query for id on server, compared it with id on client, if id on server is bigger, means data on server has changed, then app will query server for these data, download it, then update id on client with id on server
4) not using timestamp on every records, but using auto increment int, cause there are three local timezone in my country. If using timestamp, I will get another headache to make sure same timezone of every client with timezone on server.
Ok guys, thanks a lot for your suggestions, helps, ideas,etc.
Since I am not continue using multi threads with sqlite, discussion stop here.
But I am not regrets, caused got new knowledge from you all and know more about threading,RDC and SQL non batch, etc.
Last edited:
Upvote
0
But how do you get only the updated records? I don't understand. Your solution seems only valid for a complete table to download.
To avoid dealing with timezones on the server, all your timestamps have to be stored in UTC. And all your queries should be done with a timestamp in UTC as well. Databases and SQL have functions to do this conversion for you (when not done by default). You can also use the number of milliseconds since epoch if you prefer because it's what prevails in the Unix world (it's what returns DateTime.Now in B4A).
Last edited:
Upvote
0
Similar Threads
- Question
Android Question
[SOLVED] B4A v9.30 BETA dont work with MultiDex:True
- Locked
- Article
Android Tutorial
android.jar / targetSdkVersion / minSdkVersion