Android Question [SOLVED] Out of Memory error trying to load a large CSV file

[IndieDev]

Hi,

I'm trying to load a CSV file from:
https://data.covid19india.org/csv/latest/districts.csv

    Dim j As HttpJob
    j.Initialize("", Me)
    j.Download("https://data.covid19india.org/csv/latest/districts.csv")
    Wait For (j) JobDone(j As HttpJob)
    
    If j.Success Then
        File.WriteString(Starter.rp.GetSafeDirDefaultExternal("csvinfo"), "district.csv", j.GetString)
        Display_StateNames
    Else
        ToastMessageShow("Could not load State data.", True)
    End If
    j.Release

This works fine (and fast).

    Dim table As List = su.LoadCSV(Starter.rp.GetSafeDirDefaultExternal("pkinfo"), "district.csv", ",")
    Log("Adding State names to combo box")
    Dim ColNum As Int = 1

    Dim newList As List
    newList.initialize
    For Each row() As String In table
        If newList.indexOf(row(ColNum)) = -1 Then
            newList.add(row(ColNum))
            Log("State=" & row(ColNum))
        End If
    Next
    cmb_states.cmbBox.AddAll(newList)

This is extracting the unique names (I can see the log), but after a long time it displays an OutOfMemory error.

Spoiler: Error

java.lang.OutOfMemoryError: OutOfMemoryError thrown while trying to throw OutOfMemoryError; no stack trace available

How do I get extract the unique names fast?
Want to list these names in a dropdown combobox.

The csv has 344127 lines!!

Regards.

 

[IndieDev]

Sorry to trouble you all. ?

Just tried "clean project" and ran it again.
Worked fine and fast!

Just one small query here.
The list is generated fine, but how do I sort the list alphabetically before adding to the combobox?

(Will check the search option meanwhile...)

Regards.

 

[Erel]

There are exactly 0 reasons to use rp.GetSafeDirDefaultExternal.
Use File.DirInternal. Experts can use XUI.DefaultFolder (both are exactly the same).

Test is in release mode if you encounter performance issues.
You can create a custom type and then sort it with List.SortType, however in this case it will be better to asynchronously insert the data to a SQLite db and then asynchronously get the sorted data. This way the UI will not freeze.

 

[IndieDev]

Found it.

newList.Sort(True)

before adding the list to the combobox.

 

[IndieDev]

There are exactly 0 reasons to use rp.GetSafeDirDefaultExternal.
Use File.DirInternal. Experts can use XUI.DefaultFolder (both are exactly the same).

Test is in release mode if you encounter performance issues.
You can create a custom type and then sort it with List.SortType, however in this case it will be better to asynchronously insert the data to a SQLite db and then asynchronously get the sorted data. This way the UI will not freeze.

Thanks, Erel.

Was actually in 2 minds regarding using GetSafeDirDefaultExternal.
Will try now using File.DirInternal.

Once I get the code running, will checkout inserting into SQLite db.
(the UI is freezing for a few seconds now, actually)