Android Question Multi core processing in Android

[incendio]

Today's, phones / tablets, is common to have more than 1 core.
When app doings some processing, is it using all cores or only 1 core used? And who's controlling how many cores used by app, Android OS, or developer?

 

[incendio]

You can ask the OS to create threads for you. You don't have any control over the cores. The OS is responsible for that.

How to ask OS to create threads? With threading library?

Note that tou need to be careful with SQLite database and multithreaded apps.

Have you encountered any performance issue?

I am still new in sqlite features, and when using multi threads, I divided data into available threads, for example, there are 100 records & 4 threads, I assigned each threads with 25 records.

So far, performance is better, with 4 threads, at least 6-7 times faster.

The bottle neck is when downloading data with RDC, no matter how many thread created, finished time still the same with 1 thread.
Is this a characteristics of httputtils or connection pooling by jetty?

 

[Erel]

Inserting 100 records to a database should be a very quick operation if done correctly.

Can you post the code?

As I wrote above SQLite support for multithreading is very delicate. You can easily corrupt your database.

HttpUtils2 sends each request on a different thread. You cannot make it faster by adding more threads.

 

[incendio]

100 records is just for an example. Actual records started with about 20000 records and growing.

I can post a codes latter when on my CPU, right now, browsing via tablet.

 

[Informatix]

100 records is just for an example. Actual records started with about 20000 records and growing.

I can post a codes latter when on my CPU, right now, browsing via tablet.

Do you really need to retrieve 20000 records each time? Or just the first time? What's the volume of data? And what's the accepted duration to write these records to the dabase? Because, enclosed in a transaction, that should be done in less than 30 seconds.

 

[incendio]

Do you really need to retrieve 20000 records each time? Or just the first time? What's the volume of data? And what's the accepted duration to write these records to the dabase? Because, enclosed in a transaction, that should be done in less than 30 seconds.

First time, data from server (+/-20000 records) uploaded to devices. By the time goes by, could be daily/weekly/monthly, data from server gets update,add, and delete(rarely). This new data must upload again to devices.

I have to alternatives to upload this data
1) dirty but simpler, delete all data from devices and replaced it with new data from server (which is currently I choose)

2) complex but efficient, upload only new and modified data from server to devices

Alternative 2 is complex cause must keep a log on server about new data, inserted data & deleted data and then record every device id and keep tracks which device id has updated its data and which one has not. Also must consider device that is not updated its data, but server already update/delete the data. If running well, device could be more than 250.

Volume of data, for first time is about 20000 rows x 8 columns. Column data type, 3 int, 1 char(3), 2 varchar(60) & 1 varchar(4).

I always do sql operations within transaction. With current data, single thread, time to write records is about 18 secs. If speed is linear, than when data reach above 30000, it will took more than 30 secs. But speed is also depend on how fast internal SD Card write & read and how fast cpu on device.

With multi thread, performance increased, see the attached diagram, but as Erel said, that could be corrupted database, I began to consider alternative 2.

 

[thedesolatesoul]

Atleast it seems your downloads are not being threaded for some reason. Without seeing code it is impossible to say, also I am not familiar with RDC.
However, since all HttpUtils2 will be raising the same JobDone back on to the main thread, you need to ensure that JobDone delegates the tasks off the Main thread, and the main thread remains empty.

 

[incendio]

Can you post the code?

Atleast it seems your downloads are not being threaded for some reason. Without seeing code it is impossible to say, also I am not familiar with RDC.
However, since all HttpUtils2 will be raising the same JobDone back on to the main thread, you need to ensure that JobDone delegates the tasks off the Main thread, and the main thread remains empty.

You were right, downloads is was not threaded, in runs in sequence.
Here are my codes to create 4 request to RDC

Sub GetData
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data1"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data1")

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data2"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data2")

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data3"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data3")

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data4"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data4")
End Sub

Sub JobDone(Job As HttpJob)
   If Job.Success = true Then
      If Job.JobName = "DBRequest" Then
         Dim result As DBResult = reqM.HandleJob(Job)
   
         If result.Tag = "get_data1" OR  & _
            result.Tag = "get_data2" OR & _
            result.Tag = "get_data3" OR & _
            result.Tag = "get_data4" Then
            Log(DateTime.Time(DateTime.now))
            UpdateData(result)
      End If
   End If
   Job.Release
End Sub

Sub UpdateData(result As DBResult)
  Try
     Private ValC1,ValC2,ValC3,ValC4 As String
     Private ValI1,Val2,Val3 As Int
     For Each records() As Object In result.Rows
        ValI1 = records(6)   
        ValI2 = records(0)
        ValC1 = records(1)
        ValC2 = records(2)
        ValC3 = records(3)
        ValC4 = records(4)
        ValI3 = records(5)
   
        SQL.AddNonQueryToBatch("insert into table(Field1,
Field2,Field3,Field4,Field5,Field6,Field7) values(?,?,?,?,?,?,?)",Array As Object(ValI1,ValI2,ValC1,ValC2,ValC3,ValC4,ValI3))
   
     Next
     SQL.ExecNonQueryBatch("InsData")
  Catch
     Log(LastException)
  End Try
End Sub

Sub InsData_NonQueryComplete(Success As Boolean)
   Log(DateTime.Time(DateTime.now))
End Sub

There is no ExecNonQueryBatch in RDC, so I don't know how to threaded this download.

Also, there is no transaction when using SQL.ExecNonQueryBatch cause I read, it is already handle internally. When using 1 thread and SQL.ExecQuery, I used it within transaction.

 

[DonManfred]

You were right, downloads is was not threaded, in runs in sequence.

No, they are NOT in order

Note:

Under the hood DBRequestManager creates a HttpJob for each request. The job name is always "DBRequest".

and httpjobs runs parallel and in fact your code

Sub GetData
    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data1"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data1")

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data2"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data2")

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data3"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data3")

    Dim cmd As DBCommand
    cmd.Initialize
    cmd.Name = "get_data4"
    DBRequestManager.ExecuteQuery(cmd, 0, "get_data4")
End Sub

Starts 4 jobs and run them all in background.
for ex. get_data4 could return first.

I suppose that you need to take care of the order if that is important to you

 

[incendio]

You were right about the order, some time the last one could finished first.
But I think, it was not runs in parallel.

I got something like these from the log
10:00:30
handling job xxxx
10:00:33
handling job xxxx
10:00:37
handling job xxxx
10:00:40
handling job xxxx

From the log, each seem that every job executed after previous job finished.

 

[DonManfred]

From the log, each seem that every job executed after previous job finished.

I dont know what the requestmanager does. Maybe he take care of the order. But maybe it is just accidental that the order is correct.

I DONT know and not worked with dbrequestmanager before. But if it just fires 4 httpjobs like i would expect (after read the sidenotes about httpjob in the docu) then the order is not garanteed.

Maybe we need @Erel ´s wisdom to answer this

 

[DonManfred]

Try to create a get_data5 which (server side) just return a special id or something. Not a list of rows. execute a fast query there
So the get_data5 should be answer faster than the others and the result should be return faster in JobDone

 

[incendio]

I dont know what the requestmanager does. Maybe he take care of the order. But maybe it is just accidental that the order is correct.

I DONT know and not worked with dbrequestmanager before. But if it just fires 4 httpjobs like i would expect (after read the sidenotes about httpjob in the docu) then the order is not garanteed.

Maybe we need @Erel ´s wisdom to answer this

You misunderstood, may be my lack of english made me not explain clearly, sorry.
You were right about order, it is not guaranteed that Job #3 will finished before Job # 4. Sometime Job order #4 could be finished before Job order #3.

What I mean here is, jobs were not executed in parallel. If it runs in parallel, I would expect interval to execute between jobs will be the same or not much different. May be the log would be something like these :
10:00:30
Handling job #3
10:00:30
Handling job #2
10:00:31
Handling job #1
10:00:32
Handling job #4

 

[Erel]

Which database server are you using?

 

[incendio]

Firebird.

 

[Erel]

Maybe the database server handles the requests synchronously.

You can see performance numbers of a server app created with B4J and MySQL: http://www.b4x.com/android/forum/threads/37502/#content
It is similar to RDC.

 

[Informatix]

Alternative 2 is complex cause must keep a log on server about new data, inserted data & deleted data and then record every device id and keep tracks which device id has updated its data and which one has not. Also must consider device that is not updated its data, but server already update/delete the data. If running well, device could be more than 250.

Volume of data, for first time is about 20000 rows x 8 columns. Column data type, 3 int, 1 char(3), 2 varchar(60) & 1 varchar(4).

1,4 Mb for the whole database is not too heavy to consider a complete download each time, but if your user has a very slow connection or a limited volume of data per month, he/she won't be happy to have to download everything each time he/she uses your app.
In your scenario (multiple readers for a remote database), the option 2 is the way to go and is not as complex as you think:
1) Add a timestamp to each record and query from the client only the records that are newer than the latest timestamp retrieved with the last download.
2) Create a new table on the server storing the ID of all deleted records with the timestamp of their deletion (this table could be filled automatically with a trigger). Do the same kind of query as above to know the records to delete.

 

[incendio]

Maybe the database server handles the requests synchronously.

You can see performance numbers of a server app created with B4J and MySQL: http://www.b4x.com/android/forum/threads/37502/#content
It is similar to RDC.

This database server is also used in LAN and I am pretty sure that it is set for asynchronously, but I will check again its JDBC.
Is there any settings in I should also check in file c3p0.properties (file from jetty) to make sure that it handle request in asynchronously mode?

 

[Erel]

Is there any settings in I should also check in file c3p0.properties

If you haven't changed it then it creates a pool of 15 connections.

 

[incendio]

1,4 Mb for the whole database is not too heavy to consider a complete download each time, but if your user has a very slow connection or a limited volume of data per month, he/she won't be happy to have to download everything each time he/she uses your app.
In your scenario (multiple readers for a remote database), the option 2 is the way to go and is not as complex as you think:
1) Add a timestamp to each record and query from the client only the records that are newer than the latest timestamp retrieved with the last download.
2) Create a new table on the server storing the ID of all deleted records with the timestamp of their deletion (this table could be filled automatically with a trigger). Do the same kind of query as above to know the records to delete.

Thanks for your suggestion. Considering that there is a potential corrupted sqlite database in devices, which is a more big problems, it seem that this the only solution.

I will modified this idea a bit :
1) create new table on server & client, consist only 1 field,type incremental int id
2) every time records on server get update/insert/delete, id on server's table increased by 1
3) when users login to app, app query for id on server, compared it with id on client, if id on server is bigger, means data on server has changed, then app will query server for these data, download it, then update id on client with id on server
4) not using timestamp on every records, but using auto increment int, cause there are three local timezone in my country. If using timestamp, I will get another headache to make sure same timezone of every client with timezone on server.

Ok guys, thanks a lot for your suggestions, helps, ideas,etc.
Since I am not continue using multi threads with sqlite, discussion stop here.

But I am not regrets, caused got new knowledge from you all and know more about threading,RDC and SQL non batch, etc.

 

[Informatix]

I will modified this idea a bit :
1) create new table on server & client, consist only 1 field,type incremental int id
2) every time records on server get update/insert/delete, id on server's table increased by 1
3) when users login to app, app query for id on server, compared it with id on client, if id on server is bigger, means data on server has changed, then app will query server for these data, download it, then update id on client with id on server
4) not using timestamp on every records, but using auto increment int, cause there are three local timezone in my country. If using timestamp, I will get another headache to make sure same timezone of every client with timezone on server.

But how do you get only the updated records? I don't understand. Your solution seems only valid for a complete table to download.
To avoid dealing with timezones on the server, all your timestamps have to be stored in UTC. And all your queries should be done with a timestamp in UTC as well. Databases and SQL have functions to do this conversion for you (when not done by default). You can also use the number of milliseconds since epoch if you prefer because it's what prevails in the Unix world (it's what returns DateTime.Now in B4A).