Android Question JSONParser nextArray fail with big JSON string

[Gianni Sassanelli]

Hi
i need to convert a JSON string to list but nextArray function fail if JSON string is too big

Alert before unzip the atthached file, rename it as explained and open with 7zip:
jsonString_001.zip to jsonString.zip.001
jsonString_002.zip to jsonString.zip.002

my code is:

Dim JSon         As JSONParser
Dim JsonString AS String
JsonString = "big json string see attached file"
Dim JSLs AS List
JSLs.Initialize
JSLs = JSon.nextArray  'if Json > 2Mb if fail'
' any idea ???

 

[Kiffi]

no problem under B4J

Dim JSon As JSONParser

Dim JsonString As String
JsonString = File.ReadString("myPath", "jsonString.txt")
JSon.Initialize(JsonString)
Dim JSLs As List
JSLs.Initialize
JSLs = JSon.nextArray
Log(JSLs.Size)

Log-Output: 16459

 

[Erel]

You should always post the error message.

Works fine here with this code:

Dim jp As JSONParser
jp.Initialize(File.ReadString(File.DirAssets, "jsonString.txt"))
Dim data As List = jp.NextArray
Log(data.Size)

 

[Gianni Sassanelli]

You should always post the error message.

the error is in attached file

 

[Erel]

Right click on the logs and post it as text.

 

[Gianni Sassanelli]

HI Erel I'm try to solve splitting the JsonString in more parts
in this case the JP.NextArray work fine.
but i have the problem pushing data in List when list.Size is about more of 200000

note that i don't read JSonString from file but i get it from a webserver
Json string is fine and i write it on file of my device but the problem is where id pushing it in a list

My procedure download data from a web server and pushing data into SQLLITE DB of device.
I do this step
Download data from WS into JsonString
Convert it to List
Scan the list and INSERT or UPDATE SQLite DB

normally this process work fine but the problem is initialization db when the data to syncronize is big

can you have any suggestion form me
thank's


the following is log
------------------------




Logger connesso a: Daishin DS9
--------- beginning of /dev/log/main
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Activity (user_logon) Create, isFirst = true **
** Activity (user_logon) Resume **
** Activity (user_logon) Pause, UserClosed = true **
** Activity (home_2019) Create, isFirst = true **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (impostazioni) Create, isFirst = true **
** Activity (impostazioni) Resume **
** Activity (impostazioni) Pause, UserClosed = true **
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (impostazioni) Create, isFirst = false **
** Activity (impostazioni) Resume **
** Activity (impostazioni) Pause, UserClosed = true **
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Activity (user_logon) Create, isFirst = true **
** Activity (user_logon) Resume **
** Activity (user_logon) Pause, UserClosed = true **
** Activity (home_2019) Create, isFirst = true **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (sincro_dati) Create, isFirst = true **
** Activity (sincro_dati) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
A 11:12:57
A 11:18:17
A 11:21:21
A 11:26:48
java.lang.OutOfMemoryError: [memory exhausted]
at dalvik.system.NativeStart.main(Native Method)
** Activity (sincro_dati) Pause, UserClosed = true **
** Activity (home_2019) Resume **
** Activity (home_2019) Pause, UserClosed = true **
Copying updated assets files (1)
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Activity (main) Pause, UserClosed = true **
** Activity (user_logon) Create, isFirst = true **
** Activity (user_logon) Resume **
** Activity (user_logon) Pause, UserClosed = true **
** Activity (home_2019) Create, isFirst = true **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Resume **
mobile data state: DISCONNECTED wifi_on: 1 server ip: 192.168.0.37
wifi ip: 192.168.0.37
** Activity (home_2019) Pause, UserClosed = false **
** Activity (sincro_dati) Create, isFirst = true **
** Activity (sincro_dati) Resume **
*** Service (httputils2service) Create ***
** Service (httputils2service) Start **
A 11:39:14
java.lang.OutOfMemoryError
at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:88)
at java.lang.StringBuilder.<init>(StringBuilder.java:95)
at anywheresoftware.b4a.debug.RDebugUtils.concat(RDebugUtils.java:100)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:735)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:357)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:260)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:175)
at anywheresoftware.b4a.BA.raiseEvent(BA.java:171)
at anywheresoftware.b4a.shell.DebugResumableSub$RemoteResumableSub.resume(DebugResumableSub.java:22)
at anywheresoftware.b4a.keywords.Common$13.run(Common.java:1680)
at android.os.Handler.handleCallback(Handler.java:733)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
at dalvik.system.NativeStart.main(Native Method)
** Activity (sincro_dati) Pause, UserClosed = false **
Error occurred on line: 1030 (globale)
java.lang.NullPointerException
at org.json.JSONTokener.nextCleanInternal(JSONTokener.java:116)
at org.json.JSONTokener.nextValue(JSONTokener.java:94)
at anywheresoftware.b4a.objects.collections.JSONParser.NextArray(JSONParser.java:60)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:735)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:357)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:260)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:144)
at it.techservice.tpick.sincro_dati.onPause(sincro_dati.java:271)
at android.app.Activity.performPause(Activity.java:5335)
at android.app.Instrumentation.callActivityOnPause(Instrumentation.java:1233)
at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3048)
at android.app.ActivityThread.performPauseActivity(ActivityThread.java:3017)
at android.app.ActivityThread.handlePauseActivity(ActivityThread.java:2995)
at android.app.ActivityThread.access$1000(ActivityThread.java:135)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1207)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:136)
at android.app.ActivityThread.main(ActivityThread.java:5017)
at java.lang.reflect.Method.invokeNative(Native Method)
at java.lang.reflect.Method.invoke(Method.java:515)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:785)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:601)
at dalvik.system.NativeStart.main(Native Method)
An error occurred:
(Line: 1032) For x = 0 To partialLS.Size - 1
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer

 

[DonManfred]

Limit the requests to the webapi to send x results. Build your own paginating system to iterate trough all then in steps of 1000 for ex.

You device simple run out of memory.

What webapi are you using? Post a link to the Api Documentation.

 

[Gianni Sassanelli]

Yes Manfred,
the problem is it but i don't have the right to limit the response and must build a solution on device.
this is a border line case but unfortunately occurs

 

[DonManfred]

I was´nt able to unzip the json you posted (tried two different 7-zip versions).

If it is a json with a simple result then you can try to parse it by yourself without using jsonparser library.
I mean maybe by using java and the GSON library.
I found this link which should be useful
http://www.acuriousanimal.com/2015/10/23/reading-json-file-in-stream-mode-with-gson.html
It require some javaknowledge but it should be doable using inlinejava i guess.

As far as i can see the json (could not get it unipped correctly as mentioned).
It starts with

[
{"cdcf":"C000001",
"cf_descrizione":"CLIENTE TEST (Prove non Cancellare)",
"indirizzo":"Indirizzo",
"localita":"Bari",
"cd_provincia":"BA",
"cd_nazione":"IT",
"cap":"70100",
"telefono":"111111111111",
"email":"",
"is_obsoleto":false
},
{"cdcf":"F000002",
"cf_descrizione":"SISTEMA S.N.C. DI CHIARIELLO \u0026 SANTO",
"indirizzo":"C.SO GARIBALDI, 192/196",
"localita":"BARLETTA",
"cd_provincia":"BT",
"cd_nazione":"IT",
"cap":"76121",
"telefono":"883531216",
"email":"",
"is_obsoleto":false
}
,

It looks like the struct is repeating and not so complicated.

Worth a try parsing it in streammode (see link above).

Can you upload the json to dropbox or something similar? So i can try it.

 

[Gianni Sassanelli]

Hi Manfred,

I solved splitting json string into multiple files of 1 MB that
Every file contain about 5000 record and i parse one file for time
thank's you

i attach my example code

Dim JSonString as String
JSonString = getJsonfromWS
Dim nStep, lenStep, EndCut, JSStringLen As Int
Dim fileName,partialJS As String
JSStringLen = SF.LEN(JSonString)
lenStep = 1048576
nStep = Ceil(JSStringLen/ lenStep)
For i = 1 To nStep
   EndCut = JsonString.IndexOf2("}",lenStep) + 1   
   If EndCut = 0 Then
      EndCut = sf.Len(JsonString)
   End If
   partialJS  = sf.Mid(JsonString,1,EndCut )
   fileName = "partialJS_" & sf.Pad(i,"0",3,False) &".txt"
   File.WriteString(File.DirRootExternal, fileName,partialJS)
   JsonString = sf.MidS(JsonString,EndCut+1)       
Next