B4J Question Reading big csv-file: OutOfMemoryError

[Kiffi]

Hello,

i wrote a simple code to read a big csv-file line-by-line following this example:

http://www.b4x.com/android/forum/threads/parsing-huge-text-files.34923/

Here is my code:

  Dim myTextReader As TextReader

    Try
        ' big.csv (266 MB ~ 790.000 lines)
        myTextReader.Initialize(File.OpenInput("D:\data\", "big.csv") )
        Dim LineCounter As Int = 0
        Dim line As String = myTextReader.ReadLine
        Do While line <> Null
            LineCounter = LineCounter + 1
            line = myTextReader.ReadLine
            If LineCounter Mod 10000 = 0 Then
                Log("Line: " & NumberFormat(LineCounter, 0, 0))
            End If
        Loop
        myTextReader.Close
    Catch
        Log("Error --> " & LastException.Message)
        If myTextReader.IsInitialized Then
            myTextReader.Close
        End If
    End Try

after ~ 270.000 lines i get this error:

Program started.
Line: 10,000
[..]
Line: 270,000
Exception in runnable
java.lang.OutOfMemoryError: Java heap space

I thought, reading a file line-by-line wouldn't increase the memory consumption.

What did I do wrong?

Thanks in advance & Greetings ... Kiffi (Windows Vista Business / 32 Bit / 4 GB RAM)

 

[Erel]

Maybe there is a huge line somewhere in your file? What is this file?

 

[Kiffi]

Hello Erel,

thanks for your fast reply!

Maybe there is a huge line somewhere in your file?

unfortunately not. In the meantime i have generated another test-csv (~380 MB) with 1.000.000 of dummy-lines:

Field1;Field2;Field3;[...];Field50

Same error.

I wonder, why the memory-consumption is increasing during the runtime (see attached image)

Can it be, that there is a memory leak?

Thanks again & Greetings ... Kiffi (B4J V1.05)

 

[Erel]

In my example I parsed a 3.7gb. If you can, create a small project that creates the CSV file and then parses it. I will test it.

 

[Kiffi]

Very strange...

On my computer at office (Vista / 32 Bit / 4 GB) i am able to read ~ 200.000 lines (out of 1.000.000) and get a socket(!)-error.

On my computer at home (Win7 / 64 Bit / 4 GB) i can read ~ 700.000 lines and get a out-of-memory exception.

I will test it.

Thanks for your kind support! The code is attached to this posting.

Greetings ... Kiffi

 

[Erel]

I forgot to ask, are you testing it in Release mode? Such programs should be tested in Release. The debugger is less efficient.

 

[Kiffi]

Hello Erel,

I forgot to ask, are you testing it in Release mode? Such programs should be tested in Release. The debugger is less efficient.

now i have tested in release mode and: WHOW! It works! All lines are read and i get no error. And it's amazing fast. [ThumbsUp]

Thanks again & Greetings ... Kiffi