Android Question Split a db file

[pesquera]

Hi,
How can I split one SQLite db file into fragments of 1024 bytes?
Thanks in advance

 

[Erel]

Why does it matter that it is a db file? Do you want each fragment to be a working db file as well?

 

[pesquera]

No, it's not important to be a working db file each part

Let's me explain:
I'm using your fantastic CloudKVS Class, it is very useful to me.. I do backup/restore from some parameters and also files, it's perfect

I can do that with all the parameters and small txt/db files correctly.. but, with one db file I have an issue when the file is too large
Seems that doing a put (ckvs) with a large db file corrupts the db cursor, when trying to get the record (rs.Get..)
It's related to this post: https://www.b4x.com/android/forum/threads/ckvs-put-value-sqlite-db.66318
(sorry if this is considered a cross-posting)

I've tried compressing the file without luck.. so, I'm thinking about splitting the db file and do a put (ckvs) with each part of the file
Later, at restore time, will do the inverse way (get each part and combine then to one db file)

I hope to be clear about my issue.. thanks very much for your help!

 

[Erel]

I don't think that it is related to the size. I've just tested it with a 9mb image and it was transferred properly.

Are you closing the database before you add it to the store?

 

[pesquera]

yes, I am
This is the code (cleaned)

Starter.dbSQL.Close
File.Copy(File.DirInternal,"agendame.turnos.db",File.DirDefaultExternal,"agendame.turnos.db")
InitializeDB(Starter.dbSQL,File.DirInternal,"agendame.turnos.db",False,Application.VersionCode)
Dim fileSync() As Byte=Bit.InputStreamToBytes(File.OpenInput(File.DirDefaultExternal,"agendame.turnos.db"))   
Starter.ckvs.Put(Starter.ckvsUser,"localData_Agendame.Turnos.db",fileSync)

Public Sub InitializeDB(dbSQL As SQL, DB_Location As String, DB_Fullname As String, CreateAlways As Boolean,Version As String)
Try
    If Not(File.Exists(DB_Location, DB_Fullname)) Or CreateAlways=True Then
        File.Copy(File.DirAssets, DB_Fullname, DB_Location, DB_Fullname)
    End If
    If Not(dbSQL.IsInitialized) Then
       dbSQL.Initialize(DB_Location, DB_Fullname, False)
       dbSQL.BeginTransaction
       Try
           dbSQL.ExecNonQuery("DELETE FROM Search") 
           dbSQL.ExecNonQuery("update SQLite_Sequence set seq=0 where name='Search'")
           dbSQL.ExecNonQuery("DELETE FROM TT_Grid") 
           dbSQL.ExecNonQuery("update SQLite_Sequence set seq=0 where name='TT_Grid'")
           dbSQL.TransactionSuccessful
       Catch
              log(LastException)
       End Try      
       dbSQL.EndTransaction
    End If
Catch
       log(LastException)
End Try       
End Sub

The issue does not happen when the size of the db is around 1Mb
Should be another problem, but.. I can't figure it out
May be is my db definition? If you want, I can send it by email (because of the limit to upload the file here)

The error occurs on the Sub HandleQueue, on the rs.Get method

Error occurred on line: 123 (ClientKVS)
java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
    at android.database.CursorWindow.nativeGetLong(Native Method)
    at android.database.CursorWindow.getLong(CursorWindow.java:524)
    at android.database.AbstractWindowedCursor.getLong(AbstractWindowedCursor.java:75)
    at anywheresoftware.b4a.sql.SQL$CursorWrapper.GetLong(SQL.java:373)
    at java.lang.reflect.Method.invoke(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:372)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:702)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:336)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:246)
    at java.lang.reflect.Method.invoke(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:372)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:134)
    at anywheresoftware.b4a.BA$2.run(BA.java:328)
    at android.os.Handler.handleCallback(Handler.java:739)
    at android.os.Handler.dispatchMessage(Handler.java:95)
    at android.os.Looper.loop(Looper.java:145)
    at android.app.ActivityThread.main(ActivityThread.java:5832)
    at java.lang.reflect.Method.invoke(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:372)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1399)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1194)

 

[DonManfred]

You are closing it. Then you copy it. Then you initialize it again and put it into the kvs? So it it not closed???

 

[pesquera]

The put is from File.DirDefaultExternal while later is initialized from File.DirInternal
The close is relevant before the copy
Anyway, I moved InitializeDB to the end of the block code and do happen the same issue
Thanks for your help

 

[Erel]

Please download another copy of the B4A example: https://www.b4x.com/android/forum/threads/b4x-cloudkvs-synchronized-key-value-store.63536/#content
Add the database to the store and see whether you are getting this error.

What is the database size?

 

[pesquera]

db size is about 2.5 Mb, it has 5 tables but only 2 tables have data (integer, varchar, date, time and blob fields)

I noticed now that on B4J Client the issue does not happen, the problem seems to be only on B4A

I've followed your suggestion, and put the db file into Assets so added these lines for testing:

Dim fileSync() As Byte=Bit.InputStreamToBytes(File.OpenInput(File.DirAssets,"dbFailed.db"))   
Starter.ckvs.Put(user,"dbFailed.db",fileSync)

here is the log with the error:

** Service (starter) Create **
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Service (httputils2service) Create **
** Service (httputils2service) Start **
Class not found: b4j.cloudkvs.server.main$_item, trying: b4a.example.main$_item
**** User: u1 ****
Error occurred on line: 261 (ClientKVS)
java.lang.IllegalStateException: Couldn't read row 0, col 0 from CursorWindow.  Make sure the Cursor is initialized correctly before accessing data from it.
    at android.database.CursorWindow.nativeGetString(Native Method)
    at android.database.CursorWindow.getString(CursorWindow.java:439)
    at android.database.AbstractWindowedCursor.getString(AbstractWindowedCursor.java:51)
    at anywheresoftware.b4a.sql.SQL$CursorWrapper.GetString(SQL.java:355)
    at b4a.example.clientkvs._getall(clientkvs.java:488)
    at b4a.example.clientkvs._utilprintdata(clientkvs.java:72)
    at b4a.example.main._ckvs_newdata(main.java:451)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:515)
    at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:702)
    at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:339)
    at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:246)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:515)
    at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:134)
    at anywheresoftware.b4a.BA$2.run(BA.java:328)
    at android.os.Handler.handleCallback(Handler.java:733)
    at android.os.Handler.dispatchMessage(Handler.java:95)
    at android.os.Looper.loop(Looper.java:146)
    at android.app.ActivityThread.main(ActivityThread.java:5602)
    at java.lang.reflect.Method.invokeNative(Native Method)
    at java.lang.reflect.Method.invoke(Method.java:515)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1283)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1099)
    at dalvik.system.NativeStart.main(Native Method)

Thanks Erel for the help.. but, I've already solved it changing my code.. reading just few fields from all records of the db, and later I fill the rest of the fields on the server side
I've tested again and put the error just in case that will be helpful
Please let me know if needed some additional info

 

[Erel]

The problem is in the Android implementation of SQLite. There seems to be a maximum size limit of 1mb: http://stackoverflow.com/questions/5406429/cursor-size-limit-in-android-sqlitedatabase

Based on my tests with Android 6.0 the limit is much larger on newer versions so I wasn't able to reproduce it when I tested it with a 9mb image.

 

[pesquera]

Thanks for that info!