Android Question [SOLVED] Out of Memory error trying to load a large CSV file

IndieDev

Active Member
Licensed User
Hi,

I'm trying to load a CSV file from:
https://data.covid19india.org/csv/latest/districts.csv

Downloading from URL and saving to local folder:
    Dim j As HttpJob
    j.Initialize("", Me)
    j.Download("https://data.covid19india.org/csv/latest/districts.csv")
    Wait For (j) JobDone(j As HttpJob)
    
    If j.Success Then
        File.WriteString(Starter.rp.GetSafeDirDefaultExternal("csvinfo"), "district.csv", j.GetString)
        Display_StateNames
    Else
        ToastMessageShow("Could not load State data.", True)
    End If
    j.Release
This works fine (and fast).

Then I'm trying to get the unique names using the following code:
    Dim table As List = su.LoadCSV(Starter.rp.GetSafeDirDefaultExternal("pkinfo"), "district.csv", ",")
    Log("Adding State names to combo box")
    Dim ColNum As Int = 1

    Dim newList As List
    newList.initialize
    For Each row() As String In table
        If newList.indexOf(row(ColNum)) = -1 Then
            newList.add(row(ColNum))
            Log("State=" & row(ColNum))
        End If
    Next
    cmb_states.cmbBox.AddAll(newList)

This is extracting the unique names (I can see the log), but after a long time it displays an OutOfMemory error.
java.lang.OutOfMemoryError: OutOfMemoryError thrown while trying to throw OutOfMemoryError; no stack trace available

How do I get extract the unique names fast?
Want to list these names in a dropdown combobox.

The csv has 344127 lines!!

Regards.
 

IndieDev

Active Member
Licensed User
Sorry to trouble you all.

Just tried "clean project" and ran it again.
Worked fine and fast!

Just one small query here.
The list is generated fine, but how do I sort the list alphabetically before adding to the combobox?

(Will check the search option meanwhile...)

Regards.
 
Last edited:
Upvote 0

Erel

B4X founder
Staff member
Licensed User
Longtime User
There are exactly 0 reasons to use rp.GetSafeDirDefaultExternal.
Use File.DirInternal. Experts can use XUI.DefaultFolder (both are exactly the same).

Test is in release mode if you encounter performance issues.
You can create a custom type and then sort it with List.SortType, however in this case it will be better to asynchronously insert the data to a SQLite db and then asynchronously get the sorted data. This way the UI will not freeze.
 
Upvote 0

IndieDev

Active Member
Licensed User
Thanks, Erel.

Was actually in 2 minds regarding using GetSafeDirDefaultExternal.
Will try now using File.DirInternal.

Once I get the code running, will checkout inserting into SQLite db.
(the UI is freezing for a few seconds now, actually)
 
Upvote 0
Cookies are required to use this site. You must accept them to continue using the site. Learn more…