B4J Question Server application running on AWS is crashing

[avalle]

My server application implementing a service providing REST API, crashes from time to time.
It runs on AWS cloud instance with Linux AMI, using the usual way:

sudo nohup java -jar myserver.jar &

There's no particular code path crashing the server, and it's not systematic. Sometimes it even seems to crash during "idle" time.
This is quite annoying and I'd like the experts in the forum to recommend solutions to solve or mitigate the issue.

My suspect is that the B4J app runs out of memory, given the VPS it's pretty minimal (it's a t2.micro instance type: 1 vCPU / 1 GB RAM).
How can I monitor this?
How do I realize that the B4J app has crashed due to low memory?
Or if the VPS is running out of memory?
Does it help to assign more heap space to the B4J app?

Sorry if all these questions look basic, that's exactly my level of knowledge about Linux/Java memory management... be patient...

Thanks
Andrea

 

[DonManfred]

How could we help if you just do not post any stacktrace from the crash? We can not guess the problem/error

You can try to redirect the log-output to a file and watch the logfile after the server crashes.

 

[teddybear]

Could you cat the nohup.out?

 

[avalle]

How do I get that?
Thanks

 

[teddybear]

How do I get that?
Thanks

It should be in file nohup.out. you can find it then cat nohup.out to show the crash log.

 

[DonManfred]

How do I get that?

reading what i wrote and use the forumsearch first? ;-)

Redirect the output to a file

Name: RedirectOutput Description: This sub will redirect the program output to a file. Sub RedirectOutput (Dir As String, FileName As String) #if RELEASE Dim out As OutputStream = File.OpenOutput(Dir, FileName, False) 'Set to True to append the logs Dim ps As JavaObject...

www.b4x.com

 

[avalle]

Oh I see...
I'm not talking about exceptions occurring in the B4J app that I can view and analyze from the error logs.
I'm talking about the "sudo java" process being terminated, so clearly nothing shows up in the error logs because the application does not terminate due to an internal exception, but likely the process is terminated by the OS due to lack of memory.

 

[Quandalle]

Your line

sudo nohup java -jar myserver.jar &

The ampersand will background sudo here rather than the command being run as sudo; use the -b flag to sudo instead

I generally use the following batch file (.sh) to start a java server on Linux

echo '8234567ss_26' | sudo -S -b java -jar myserver.jar

where

• '8234567ss_26' is the sudo password
• The -b (background) option tells sudo to run the givencommand in the background
• The -S (stdin) option causes sudo to read the password from the standard input instead of the terminal device.

if you have not redirected the logs(stdout,stderr) yourself you can also add nohup before java command

 

[avalle]

Hi Quandalle, thanks for your suggestions but I don't think this is the issue.
Again, you hit my lack of experience with Linux, but I think I'm using the recommended way to launch a B4J server app according to various posts here in the Forums:

sudo nohup java -jar myserver.jar > nohup.out &

This also allows me to log all the StdOut / StdErr output to a file.

But again, my issue is not about running the server, my issue is its stability, to avoid that a crash would shut it down during normal use.

 

[Quandalle]

there is no stability problem in a solution of this type: AWS + Linux + JVM. All these components can be considered as reliable. So if your application is killed it's not a stability problem. There is a reason and a log file somewhere should have a trace of the operation that has been done .

If you have examined the StdOut/and StdErr outputs of the JVM without finding any information it means that the action comes from outside Java environnent, so from the system.

On Linux you should examine the logs (/var/log/messages ) and the kernel information (dmesg ).
For example, to find out if you can find the word "killed"

sudo dmesg -T | grep killed
sudo cat /var/log/messages | grep killed

Your machine having little memory, we can possibly suspect the management of the Overcommit memory. You should know if your application is the only one running on the machine or if other applications are likely to start at certain times.

This is because in the OverCommit Memory system, when a process requests more space, Linux will give it that space, assuming that not all processes actually use all the memory they have requested. The process will get exclusive use of the memory it requested when it actually uses it. This makes allocation fast, and can allow more memory to be allocated than is actually there.

However, once processes start using this memory, if there is not enough memory, Linux will kill a process to free up memory. The choice of which process to kill is based on a score taking into account execution time (long processes are safer), memory usage (greedy processes are less safe), and some other parameters adjustable by the system administrator. It is the OOM Killer function of the kernel that takes care of killing a process.

the pseudo files in the /proc/sys/vm directory allow to know the various parameters or to change them
for example

cd /proc/sys/vm
cat overcommit_memory

it displays a value 0,1, or 2

• with 0, or 1 the linux system uses OOM Killer if necessary
• with 2 OOM Killer is not used

(there are other overcommit_xxx files which manage the parameters)
You can also check in dmseg or in the log messages if you find traces of the string "OOM" which would tend to show the reason why the process was killed.

If the problem really comes from the overcommit (i.e. a value of 0, or 1 and traces of "kill"/"OOM" in the logs) the possible solutions are :

• Add more memory for the VM
• Don't launch other processes that could ask for memory and therefore cause the kill
• Disable the OOM Killer (value 2)
• Manage the OOM Killer parameters so that your process does not have priority

 

[Tecuma]

Disable the OOM Killer (value 2)

I am not sure if this setting is a good decision. What do you expect from Linux if it can not kill processes to operate properly?

From my experience best case: Systems than become unresponsive. If swap is configured it start to swap. You get the impression that your system is getting freeze.
Worst case: The kernel crash.

If possible configure more RAM to your system.
You may try to optimize the system that your process is not affected by OOM. There are several tutorials avaibale, e.g. this one. This should be a temporarily solution until the system has more RAM.

 

[Tecuma]

How can I monitor this?

You may check this with sar. sar is a tool to check actual performance and historical ones.
Details available in the glorious internet. E.g. this one. I would check the documentation from your Linux distribution about it. Maybe it is already installed and configured.

 

[OliverA]

How can I monitor this?
How do I realize that the B4J app has crashed due to low memory?
Or if the VPS is running out of memory?
Does it help to assign more heap space to the B4J app?

You could try running the garbage collector manually. That seems to have fixed some memory issues for @alwaysbusy (first link below). @Jmu5667 has some code in regards to this issue that has extra code for memory monitoring (second link below).

Links:

[Solved] Out of Memory Issue

My app has been functioning great for many years. July 14, 2021 it started misbehaving... This is the batch file that starts my app, which is large (24 meg).. 'cd\root 'java -Xmx1024M -jar tripinspect.jar 'pause This is the error file (log section)... 'main._appstart (java line: 279)...

www.b4x.com

Garbage collection

Hello For those of you interested in GC here is a class I have in a B4J server: Sub Class_Globals Private tmrHook As Timer Private gcTime As Long Private freq_gc As Int = 1 End Sub Public Sub Initialize ' // 3 seconds...

www.b4x.com

 

[Swissmade]

Nice problem
1e What you can do is limit the memory is using by Java.
2e A function to clear memory. see OliverA post above.
Also if there is a GUI use visual VM to see what your server is doing.