B4J Question RAM usage keep increased in a long running App

[incendio]

Hello guys,

I have an non UI B4J that runs continuously on a loop using timer.
App runs fine, but RAM usage keep increased after runs for a long time.

Tested on Ubuntu server, on started, RAM usage in Ubuntu was about 120KB, after runs for about an hour, RAM usage was about 133KB.

On windows, the behavior was the same.

Here are my codes :

'Non-UI application (console / server application)
#Region Project Attributes
    #CommandLineArgs: "3050" "10000"
    #MergeLibraries: True
#End Region
#AdditionalJar : jaybird-full-3.0.2.jar

Sub Process_Globals
    Private dbpool As ConnectionPool
    Private Sq,ExecQ1,ExecQ2 As SQL
    Private RS As ResultSet
    Private Tm As Timer
    Private Port,Interval As Int
End Sub

Sub AppStart (Args() As String)
    If Args.Length > 0 Then
        Port =  Args(0)
        Interval = Args(1)
    End If
    dbpool.Initialize( "org.firebirdsql.jdbc.FBDriver","jdbc:firebirdsql://localhost:" & Port & "/my_db?charSet=utf-8","user","pass")
    Tm.Initialize("TM",Interval)
    Tm.Enabled = True
    StartMessageLoop
End Sub

'Return true to allow the default exceptions handler to handle the uncaught exception.
Sub Application_Error (Error As Exception, StackTrace As String) As Boolean
    Return True
End Sub

Sub TM_Tick
    Private Id As Int
    Private Val As String
    Tm.Enabled = False
    Try
        Sq        = dbpool.GetConnection
        ExecQ1 = dbpool.GetConnection
        ExecQ2 = dbpool.GetConnection
        RS        = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")
        Log("OK")

        ExecQ1.BeginTransaction
        ExecQ2.BeginTransaction
        Do While RS.NextRow
            Id  = RS.GetInt("ID")
            Val = RS.GetString("VAL")
            Log(Id)
            ExecQ1.ExecNonQuery2("update or insert into rcv(id,val) values(?,?)",Array As Object(Id,Val))
            ExecQ2.ExecNonQuery("update snc_log set snc = 1 where id = " & Id)
        Loop
        ExecQ1.TransactionSuccessful
        ExecQ2.TransactionSuccessful
        RS.Close
    Catch
        Log(LastException)
    End Try
    Sq.Close
    ExecQ1.Close
    ExecQ2.Close
    Tm.Enabled = True
End Sub

Are there something wrong with my codes ? Any hint how to prevent RAM usage increase?

Thanks in advance.

 

[Erel]

Tested on Ubuntu server, on started, RAM usage in Ubuntu was about 120KB, after runs for about an hour, RAM usage was about 133KB.

133KB is a very small amount of RAM. Maybe 133MB (which is also not something that you should be bothered about).

You should let it run for a day or two and see whether it keeps increasing or not.

 

[incendio]

Sorry, you were right. It was 133MB.

Now, I have runs it about six hours, RAM usage was about 156MB & seem still increased.

This app will runs as long as server is on, 156MB on six hours looks not good.
Will try it to runs for a week.

So, no other way to prevent increase in RAM usage ?

 

[Erel]

B4J non-ui apps can run for months and more.

The first step is to understand whether there is a memory leak or not. A simple and useful test is to let it run for a long time and see whether the memory usage keeps increasing.

 

[alwaysbusy]

B4J non-ui apps can run for months and more.

I can confirm this. Isn't it just possible the connection pool is expanding/shrinking now and then depending on its needs, but the memory amount will be limited over time? (one will indeed need to check the memory consumption over a lot more time).

 

[Erel]

Yes, there many things that can cause the memory usage to increase. The garbage collector will release memory when needed.

 

[jmon]

Just wondering what happens if you write your code like this:

Sub Process_Globals
    Private dbpool AsConnectionPool
    Private Tm As Timer
    Private Port, Interval As Int
End Sub

Sub AppStart (Args() AsString)
    If Args.Length > 0Then
        Port = Args(0)
        Interval = Args(1)
    EndIf
    dbpool.Initialize( "org.firebirdsql.jdbc.FBDriver","jdbc:firebirdsql://localhost:" & Port & "/my_db?charSet=utf-8","user","pass")
    Tm.Initialize("TM",Interval)
    Tm.Enabled = True
    StartMessageLoop
End Sub

Sub TM_Tick
    Dim Id As Int
    Dim Val As String
    Dim Sq,ExecQ1,ExecQ2 As SQL
    Dim RS As ResultSet

    Tm.Enabled = False

    Sq        = dbpool.GetConnection
    ExecQ1 = dbpool.GetConnection
    ExecQ2 = dbpool.GetConnection

    Try
        RS = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")

        ExecQ1.BeginTransaction
        ExecQ2.BeginTransaction

        Log("OK")
        Do While RS.NextRow
            Id  = RS.GetInt("ID")
            Val = RS.GetString("VAL")
            Log(Id)
            ExecQ1.ExecNonQuery2("update or insert into rcv(id,val) values(?,?)", Array As Object(Id,Val))
            ExecQ2.ExecNonQuery("update snc_log set snc = 1 where id = " & Id)
        Loop
        ExecQ1.TransactionSuccessful
        ExecQ2.TransactionSuccessful
    Catch
        Log(LastException)
    End Try

    If RS.IsInitialized Then RS.close

    Sq.Close
    ExecQ1.Close
    ExecQ2.Close

    Tm.Enabled = True
End Sub

All the SQL and ResultSets are declared locally in that sub, so the garbage collector should discard them after the sub is finished? Am I right about that?
Also checking if the ResultSet has been closed properly, even if there was an error in the Try/Catch.

[edit] but as the others say, there is nothing wrong with the memory going up for a while. let it run a few days, and if you see that it keeps going up, then there could be a problem.

 

[incendio]

Just wondering what happens if you write your code like this:

Sub Process_Globals
    Private dbpool AsConnectionPool
    Private Tm As Timer
    Private Port, Interval As Int
End Sub

Sub AppStart (Args() AsString)
    If Args.Length > 0Then
        Port = Args(0)
        Interval = Args(1)
    EndIf
    dbpool.Initialize( "org.firebirdsql.jdbc.FBDriver","jdbc:firebirdsql://localhost:" & Port & "/my_db?charSet=utf-8","user","pass")
    Tm.Initialize("TM",Interval)
    Tm.Enabled = True
    StartMessageLoop
End Sub

Sub TM_Tick
    Dim Id As Int
    Dim Val As String
    Dim Sq,ExecQ1,ExecQ2 As SQL
    Dim RS As ResultSet

    Tm.Enabled = False

    Sq        = dbpool.GetConnection
    ExecQ1 = dbpool.GetConnection
    ExecQ2 = dbpool.GetConnection

    Try
        RS = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")

        ExecQ1.BeginTransaction
        ExecQ2.BeginTransaction

        Log("OK")
        Do While RS.NextRow
            Id  = RS.GetInt("ID")
            Val = RS.GetString("VAL")
            Log(Id)
            ExecQ1.ExecNonQuery2("update or insert into rcv(id,val) values(?,?)", Array As Object(Id,Val))
            ExecQ2.ExecNonQuery("update snc_log set snc = 1 where id = " & Id)
        Loop
        ExecQ1.TransactionSuccessful
        ExecQ2.TransactionSuccessful
    Catch
        Log(LastException)
    End Try

    If RS.IsInitialized Then RS.close

    Sq.Close
    ExecQ1.Close
    ExecQ2.Close

    Tm.Enabled = True
End Sub

All the SQL and ResultSets are declared locally in that sub, so the garbage collector should discard them after the sub is finished? Am I right about that?
Also checking if the ResultSet has been closed properly, even if there was an error in the Try/Catch.

[edit] but as the others say, there is nothing wrong with the memory going up for a while. let it run a few days, and if you see that it keeps going up, then there could be a problem.

I have tried this,

 Dim Sq,ExecQ1,ExecQ2 As SQL

But no avail.

I was running a test on Ubuntu in Virtual box, my real Ubuntu server will be ready around Jan 2018.
So can't test it right now.

But I have Windows server right now, will try to test it this week on this server to find out how much RAM consume after a long run.

This line of code caused RAM increase every time it was executed

RS = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")

Running only timer without any sql command didn't cause increase in RAM.

 

[OliverA]

This line of code caused RAM increase every time it was executed

Requesting system memory (memory not yet allocated to your application) is considered an expensive operation. Allocating memory internally to your application is a lot faster (no system calls need to be made). Therefore, Java is being "greedy" in taking system memory and "lazy" in releasing it back. The line of code that you give may increase your memory requirements if the garbage collector has not freed up memory yet and therefore it may be faster to get memory from the system than to have the garbage collector run through its recycling process. For the amount of RAM you are talking about though, I would not really worry about overwriting Java's build in memory management strategy. If you think that you need to influence the Java VM to release memory faster to the system itself, then this post (https://stackoverflow.com/a/23083177) over at stackoverflow may point you into the right direction. But please realize, you may adversely affect the performance of your application if it has to constantly take and release memory from the system (catch-22).

 

[Widget]

How do you know it is the B4J application that is consuming the memory? It could be the database or the OS.
For testing purposes what happens if you leave the database connections (Sq, ExecQ1, ExecQ2) connected at all times (initialize them at startup)? Does it still consume the same amount of memory?

 

[incendio]

Requesting system memory (memory not yet allocated to your application) is considered an expensive operation. Allocating memory internally to your application is a lot faster (no system calls need to be made). Therefore, Java is being "greedy" in taking system memory and "lazy" in releasing it back. The line of code that you give may increase your memory requirements if the garbage collector has not freed up memory yet and therefore it may be faster to get memory from the system than to have the garbage collector run through its recycling process. For the amount of RAM you are talking about though, I would not really worry about overwriting Java's build in memory management strategy. If you think that you need to influence the Java VM to release memory faster to the system itself, then this post (https://stackoverflow.com/a/23083177) over at stackoverflow may point you into the right direction. But please realize, you may adversely affect the performance of your application if it has to constantly take and release memory from the system (catch-22).

I will test first before overwriting this Java VM, see how much RAM consumed and will it stop at some point when garbage collector recycling its process.

Testing it on Window XP & Window 7 right now.

How do you know it is the B4J application that is consuming the memory? It could be the database or the OS.
For testing purposes what happens if you leave the database connections (Sq, ExecQ1, ExecQ2) connected at all times (initialize them at startup)? Does it still consume the same amount of memory?

Like I wrote before, the line that increased RAM every time it was executed was

RS = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")

 

[Widget]

Like I wrote before, the line that increased RAM every time it was executed was

RS = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")

1) Databases have a lot of dependencies working behind the scenes and it may not be this statement that is consuming all the memory. You need to verify it by commenting out every other DB reference as in:

Sub TM_TickPrivateIdAs IntPrivate Val AsString
 Tm.Enabled = False
Try
 Sq = dbpool.GetConnection
' ExecQ1 = dbpool.GetConnection
' ExecQ2 = dbpool.GetConnection
 RS = Sq.ExecQuery("SELECT ID, VAL from snc_log where snc = 0")
 Log("OK")

' ExecQ1.BeginTransaction
' ExecQ2.BeginTransactionDoWhile RS.NextRowId = RS.GetInt("ID")
' Val = RS.GetString("VAL")Log(Id)
' ExecQ1.ExecNonQuery2("update or insert into rcv(id,val) values(?,?)",ArrayAs Object(Id,Val))
' ExecQ2.ExecNonQuery("update snc_log set snc = 1 where id = " & Id)Loop
' ExecQ1.TransactionSuccessful
' ExecQ2.TransactionSuccessful
' RS.CloseCatchLog(LastException)EndTry
 Sq.Close
' ExecQ1.Close
' ExecQ2.Close
 Tm.Enabled = TrueEnd Sub

If it still consumes the same amount of memory then it is related to the execution of that statement.

2) Is column snc indexed? If not, SQLite will search all rows and this is inefficient and will consume memory.

3) Aren't you suppose to be executing:

ExecQ1.Endtransaction
ExecQ2.EndTransaction​

outside of the Try/Catch/Finally block of code to close the transaction? Sure the TransactionSuccessful will commit the transaction but I suspect the transaction is still technically open, especially if you get an error.

 

[incendio]

Yes, i have tried to commenting all command related to sql, so only timer runs in loop.
RAM usage almost constant.

I thought commit transaction
will also end transaction.

Will try to close transaction via code.

 

[OliverA]

Yes, i have tried to commenting all command related to sql

Your are also using c3p0 (http://www.mchange.com/projects/c3p0/) for pooling, which may introduce another layer of caching logic/memory retention logic. So try your SQL statement without using the pool and see what it does. If the same thing happens, then you need to look into configuring something in JDBC, if the issue is resolved, then it's just an artifact of using SQL pooling (and its associated caching).

 

[Erel]

My guess is that there is no issue here at all. However it is just a guess.

You need to let it run and monitor the memory usage. If it stops growing at 200mb or 2GB and doesn't continue to grow then there is no issue at all.

 

[Widget]

Yes, i have tried to commenting all command related to sql, so only timer runs in loop.
RAM usage almost constant.

I thought commit transaction
will also end transaction.

Will try to close transaction via code.

If you ever find out what was consuming all of that memory please let us know. It may help others in the future.
TIA

 

[incendio]

After run for a week, here are the result, First run : RAM usage was about 55MB, after a week, RAM usage was about 73MB.

Environment :
1) OS : Windows XP Sp3 64bit
2) Java ver 8 upd 144

There was a small increased in RAM usage. Will test again for another week.

 

[Daestrum]

An 18 meg increase after running for a week doesn't sound like you have a problem at all.

 

[incendio]

Perhaps. As long as not keep increased, it won't be a problem.

There are few things I have concern about :
1) Codes posted here is just an example. The real codes are more complex and could consumed more RAM
2) On Ubuntu in virtual machine, I noticed, RAM usage on startup was more than XP. Could be because of different java version. Ubuntu use Java 8-152, XP use 8-144. Can't tested on real Ubuntu yet, hardware not ready yet.
3) Ubuntu server has limited RAM, only 4GB & there will be others java processes running