Android Question JSONParser nextArray fail with big JSON string

Gianni Sassanelli

Active Member
Licensed User
Longtime User
Hi
i need to convert a JSON string to list but nextArray function fail if JSON string is too big

Alert before unzip the atthached file, rename it as explained and open with 7zip:
jsonString_001.zip to jsonString.zip.001
jsonString_002.zip to jsonString.zip.002

my code is:
B4X:
Dim JSon         As JSONParser
Dim JsonString AS String
JsonString = "big json string see attached file"
Dim JSLs AS List
JSLs.Initialize
JSLs = JSon.nextArray  'if Json > 2Mb if fail'
' any idea ???
 

Attachments

  • jsonString_001.zip
    500 KB · Views: 316
  • jsonString_002.zip
    266 KB · Views: 250

Kiffi

Well-Known Member
Licensed User
Longtime User
no problem under B4J
B4X:
Dim JSon As JSONParser

Dim JsonString As String
JsonString = File.ReadString("myPath", "jsonString.txt")
JSon.Initialize(JsonString)
Dim JSLs As List
JSLs.Initialize
JSLs = JSon.nextArray
Log(JSLs.Size)

Log-Output: 16459
 
Upvote 0

Gianni Sassanelli

Active Member
Licensed User
Longtime User

Attachments

  • Error.JPG
    Error.JPG
    67.4 KB · Views: 362
Upvote 0

Gianni Sassanelli

Active Member
Licensed User
Longtime User
HI Erel I'm try to solve splitting the JsonString in more parts
in this case the JP.NextArray work fine.
but i have the problem pushing data in List when list.Size is about more of 200000

note that i don't read JSonString from file but i get it from a webserver
Json string is fine and i write it on file of my device but the problem is where id pushing it in a list

My procedure download data from a web server and pushing data into SQLLITE DB of device.
I do this step
Download data from WS into JsonString
Convert it to List
Scan the list and INSERT or UPDATE SQLite DB

normally this process work fine but the problem is initialization db when the data to syncronize is big

can you have any suggestion form me
thank's


the following is log
------------------------




Logger connesso a: Daishin DS9
--------- beginning of /dev/log/main
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Activity (user_logon) Create, isFirst = true **
** Activity (user_logon) Resume **
** Activity (user_logon) Pause, UserClosed = true **
** Activity (home_2019) Create, isFirst = true **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (impostazioni) Create, isFirst = true **
** Activity (impostazioni) Resume **
** Activity (impostazioni) Pause, UserClosed = true **
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (impostazioni) Create, isFirst = false **
** Activity (impostazioni) Resume **
** Activity (impostazioni) Pause, UserClosed = true **
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Activity (user_logon) Create, isFirst = true **
** Activity (user_logon) Resume **
** Activity (user_logon) Pause, UserClosed = true **
** Activity (home_2019) Create, isFirst = true **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (sincro_dati) Create, isFirst = true **
** Activity (sincro_dati) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
A 11:12:57
A 11:18:17
A 11:21:21
A 11:26:48
java.lang.OutOfMemoryError: [memory exhausted]
at dalvik.system.NativeStart.main(Native Method)
** Activity (sincro_dati) Pause, UserClosed = true **
** Activity (home_2019) Resume **
** Activity (home_2019) Pause, UserClosed = true **
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Activity (user_logon) Create, isFirst = true **
** Activity (user_logon) Resume **
** Activity (user_logon) Pause, UserClosed = true **
** Activity (home_2019) Create, isFirst = true **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (sincro_dati) Create, isFirst = true **
** Activity (sincro_dati) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
A 11:39:14
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:88)
at java.lang.StringBuilder.<init>(StringBuilder.java:95)
at anywheresoftware.b4a.debug.RDebugUtils.concat(RDebugUtils.java:100)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:735)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:357)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:260)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:175)
at anywheresoftware.b4a.BA.raiseEvent(BA.java:171)
at anywheresoftware.b4a.shell.DebugResumableSub$RemoteResumableSub.resume(DebugResumableSub.java:22)
at anywheresoftware.b4a.keywords.Common$13.run(Common.java:1680)
at android.os.Handler.handleCallback(Handler.java:733)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
at dalvik.system.NativeStart.main(Native Method)
** Activity (sincro_dati) Pause, UserClosed = false **
Error occurred on line: 1030 (globale)
java.lang.NullPointerException
at org.json.JSONTokener.nextCleanInternal(JSONTokener.java:116)
at org.json.JSONTokener.nextValue(JSONTokener.java:94)
at anywheresoftware.b4a.objects.collections.JSONParser.NextArray(JSONParser.java:60)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:735)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:357)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:260)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at it.techservice.tpick.sincro_dati.onPause(sincro_dati.java:271)
at android.app.Activity.performPause(Activity.java:5335)
at android.app.Instrumentation.callActivityOnPause(Instrumentation.java:1233)
at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3048)
at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3017)
at android.app.ActivityThread.handlePauseActivity(ActivityThread.java:2995)
at android.app.ActivityThread.access$1000(ActivityThread.java:135)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1207)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
at dalvik.system.NativeStart.main(Native Method)
An error occurred:
(Line: 1032) For x = 0 To partialLS.Size - 1
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer
 
Upvote 0

DonManfred

Expert
Licensed User
Longtime User
Limit the requests to the webapi to send x results. Build your own paginating system to iterate trough all then in steps of 1000 for ex.

You device simple run out of memory.

What webapi are you using? Post a link to the Api Documentation.
 
Upvote 0

Gianni Sassanelli

Active Member
Licensed User
Longtime User
Yes Manfred,
the problem is it but i don't have the right to limit the response and must build a solution on device.
this is a border line case but unfortunately occurs
 
Upvote 0

DonManfred

Expert
Licensed User
Longtime User
I was´nt able to unzip the json you posted (tried two different 7-zip versions).

If it is a json with a simple result then you can try to parse it by yourself without using jsonparser library.
I mean maybe by using java and the GSON library.
I found this link which should be useful
http://www.acuriousanimal.com/2015/10/23/reading-json-file-in-stream-mode-with-gson.html
It require some javaknowledge but it should be doable using inlinejava i guess.

As far as i can see the json (could not get it unipped correctly as mentioned).
It starts with
[
{"cdcf":"C000001",
"cf_descrizione":"CLIENTE TEST (Prove non Cancellare)",
"indirizzo":"Indirizzo",
"localita":"Bari",
"cd_provincia":"BA",
"cd_nazione":"IT",
"cap":"70100",
"telefono":"111111111111",
"email":"",
"is_obsoleto":false
},
{"cdcf":"F000002",
"cf_descrizione":"SISTEMA S.N.C. DI CHIARIELLO \u0026 SANTO",
"indirizzo":"C.SO GARIBALDI, 192/196",
"localita":"BARLETTA",
"cd_provincia":"BT",
"cd_nazione":"IT",
"cap":"76121",
"telefono":"883531216",
"email":"",
"is_obsoleto":false
}
,

It looks like the struct is repeating and not so complicated.

Worth a try parsing it in streammode (see link above).

Can you upload the json to dropbox or something similar? So i can try it.
 
Last edited:
Upvote 0

Gianni Sassanelli

Active Member
Licensed User
Longtime User
Hi Manfred,

I solved splitting json string into multiple files of 1 MB that
Every file contain about 5000 record and i parse one file for time
thank's you

i attach my example code
B4X:
Dim JSonString as String
JSonString = getJsonfromWS
Dim nStep, lenStep, EndCut, JSStringLen As Int
Dim fileName,partialJS As String
JSStringLen = SF.LEN(JSonString)
lenStep = 1048576
nStep = Ceil(JSStringLen/ lenStep)
For i = 1 To nStep
   EndCut = JsonString.IndexOf2("}",lenStep) + 1   
   If EndCut = 0 Then
      EndCut = sf.Len(JsonString)
   End If
   partialJS  = sf.Mid(JsonString,1,EndCut )
   fileName = "partialJS_" & sf.Pad(i,"0",3,False) &".txt"
   File.WriteString(File.DirRootExternal, fileName,partialJS)
   JsonString = sf.MidS(JsonString,EndCut+1)       
Next
 
Upvote 0
Top