B4J Question OutOfMemory exception using AsyncStreams on very large file

[wl]

Hi,

I'm trying the B4J FTP server (https://www.b4x.com/android/forum/t...d-with-socket-and-asyncstreams.74320/#content) to try and upload a very large file (5 GB). I the process I'm getting an OutOfMemory Exception.

Is there anything I can/need to change ? I would not expect the Asynctreams will need to take the entire file in memory at one moment in time ?

    at anywheresoftware.b4a.BA.setLastException(BA.java:360)
    at b4j.example.ftpdataconnection._astream_newdata(ftpdataconnection.java:117)
    at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:91)

 

[Erel]

Is this the full error message? It happens on the server side, right?

 

[wl]

This is what I have. It seems it only occurs while transferring many files (25 GB in 10 concurrent threads in Filezilla)

java.lang.OutOfMemoryError: Java heap space
    at anywheresoftware.b4a.randomaccessfile.AsyncStreams$AIN.run(AsyncStreams.java:225)
    at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
    at anywheresoftware.b4a.randomaccessfile.AsyncStreams$AIN.run(AsyncStreams.java:225)
    at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
client: PWD

The FTP server is not using prefix mode;I f I look at the source code it is on the allocation of "data = new byte[]": nothing strange there ...

UPDATE: I did some more profiling: it seems the problem occurs when the FTP server needs to serve about 10 concurrent connections for large files (3-5 GB).

It seems as if when the client is STORing (and thus the server uses the AsyncStreams inputstream) Java keeps the entire files in memory, which for 10 connection with 3-5 GB just is too much too handle ... Is this correct ? And can this be solved ?

UPDATE2: I think I found the reason: the AsyncStreams reads from the inputstream and then passes the buffer data into a new array data (that is being passed to the NewData event). This data byte array will remain in memory until the end of the file.

I create an an own version of AsyncStreams in which I pass the buffer directly to the NewData event (with an extra param indicating the length): this does not work probably because raiseEventFromDifferentThread is asynchroneous ? If this is the case: how would I be able to handle this ?

while (working) {
                    
                    if (!prefix) {
                        int count = in.read(buffer);
                        if (count == 0)
                            continue;
                        if (count < 0) {
                            closeUnexpected();
                            break;
                        }
                        if (!working)
                            break;
                        ba.raiseEventFromDifferentThread(AsyncStreams.this, null, AsyncStreams.this, ev, true, new Object[] {buffer, count});
                    }

 

[Erel]

I create an an own version of AsyncStreams

I cannot help you with a modified version.

It seems as if when the client is STORing (and thus the server uses the AsyncStreams inputstream) Java keeps the entire files in memory, which for 10 connection with 3-5 GB just is too much too handle ... Is this correct ?

No, and it also has nothing to do with Java.

First step is to increase the max heap size. Set it to 8gb.

#VirtualMachineArgs: -Xms8g -Xmx8g

 

[wl]

Unfortunately I do not really agree... Unless I'm missing something: increasing the heap size seems not like a permanent solutions: what if more users concurrently upload even larger files (for example > 8 GB) ?

I changed the AsyncStreams code (by way of test) to call _NewData not on the main thread but on the current thread. PS: I'm aware that in this way I introduced multithreading (with all possible difficulties, race conditions etc) into the application

ba.raiseEvent(AsyncStreams.this, ev, new Object[] {data});

-> Result: I can upload any number of files (in concurrent connections) of any (large) size without ever running low on memory/heap (consistent < 100 MB where previously it ran out of memory at around 3,5 GB)

I really have the impressing that (at least my JVM) is not garbage collecting the copied buffer (copied into the variable data) before the entire upload is done, just because the _NewData event (which receive the copied buffer as the param) is queued to be run on the main thread.