B4J Question b4j server down. which log file to watch

juvanum · Nov 26, 2020

I installed my application (B4J SERVER using ABMATERIAL) on a DEBIAN GNU / Linux vps server.
every 15 days (more or less) unexpectedly my application goes down.

I looked in the nohup.out file but found nothing of significance.

What other logs should I look at to find the reason for the crash?

These are the files in the / var / log directory

thanks

Erel · Nov 26, 2020

juvanum said:
every 15 days (more or less) unexpectedly my application goes down.

What exactly happens? The process is killed? Check it with ps -ef |grep java

juvanum · Nov 26, 2020

Erel said:
What exactly happens? The process is killed? Check it with ps -ef |grep java

The process is killed.
When I check it with "ps -aux" the abj-server (alfa.jar) does not appear in the list.

I activate 2 abj-servers: alfa.jar and beta.jar
alfa.jar responds on port 5000
beta.jar responds on port 80

the app alfa.jar crashes
the app beta.jar run normally

juvanum · Nov 26, 2020

more info
file log messages.1

Nov 16 06:25:28 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24536" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 16 08:45:10 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24536" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 16 08:45:10 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="27703" x-info="http://www.rsyslog.com"] start
Nov 16 18:20:13 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="27703" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 16 18:20:14 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="31312" x-info="http://www.rsyslog.com"] start
Nov 17 03:58:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="31312" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 17 03:58:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2417" x-info="http://www.rsyslog.com"] start
Nov 17 06:25:05 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2417" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 17 14:03:34 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="2417" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 17 14:03:34 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="5996" x-info="http://www.rsyslog.com"] start
Nov 18 01:28:08 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="5996" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 18 01:28:08 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="9924" x-info="http://www.rsyslog.com"] start
Nov 18 06:25:04 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="9924" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 18 09:49:14 webapp1 kernel: [16161023.721181] VM Periodic Tas invoked oom-killer: gfp_mask=0x24201ca(GFP_HIGHUSER_MOVABLE|__GFP_COLD), nodemask=0, order=0, oom_score_adj=0
Nov 18 09:49:15 webapp1 kernel: [16161023.733987] VM Periodic Tas cpuset=/ mems_allowed=0
Nov 18 09:49:15 webapp1 kernel: [16161023.739756] CPU: 0 PID: 18459 Comm: VM Periodic Tas Not tainted 4.9.0-5-amd64 #1 Debian 4.9.65-3+deb9u2
Nov 18 09:49:15 webapp1 kernel: [16161023.749512] Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 01/01/2011
Nov 18 09:49:15 webapp1 kernel: [16161023.759096] 0000000000000000 ffffffffaef29964 ffffb3c9405c7c28 ffff932cc1e55100
Nov 18 09:49:15 webapp1 kernel: [16161023.767351] ffffffffaedfeb70 0000000000000000 0000000000000000 0000000c024213ca
Nov 18 09:49:15 webapp1 kernel: [16161023.775579] ffffffffaed84c47 00000042c1adb140 0000000000000000 0000000000000001
Nov 18 09:49:15 webapp1 kernel: [16161023.783924] Call Trace:
Nov 18 09:49:15 webapp1 kernel: [16161023.786877] [<ffffffffaef29964>] ? dump_stack+0x5c/0x78
Nov 18 09:49:15 webapp1 kernel: [16161023.792628] [<ffffffffaedfeb70>] ? dump_header+0x78/0x1fd
Nov 18 09:49:15 webapp1 kernel: [16161023.798551] [<ffffffffaed84c47>] ? get_page_from_freelist+0x3f7/0xb40
Nov 18 09:49:15 webapp1 kernel: [16161023.805445] [<ffffffffaed80bda>] ? oom_kill_process+0x21a/0x3e0
Nov 18 09:49:15 webapp1 kernel: [16161023.811819] [<ffffffffaed8085d>] ? oom_badness+0xed/0x170
Nov 18 09:49:15 webapp1 kernel: [16161023.817852] [<ffffffffaed81071>] ? out_of_memory+0x111/0x470
Nov 18 09:49:15 webapp1 kernel: [16161023.824059] [<ffffffffaed862e7>] ? __alloc_pages_slowpath+0xbb7/0xbf0
Nov 18 09:49:15 webapp1 kernel: [16161023.830972] [<ffffffffaed8651e>] ? __alloc_pages_nodemask+0x1fe/0x260
Nov 18 09:49:15 webapp1 kernel: [16161023.837988] [<ffffffffaedd6981>] ? alloc_pages_current+0x91/0x140
Nov 18 09:49:15 webapp1 kernel: [16161023.844550] [<ffffffffaed7f070>] ? filemap_fault+0x300/0x5a0
Nov 18 09:49:15 webapp1 kernel: [16161023.850689] [<ffffffffc01cd7e1>] ? ext4_filemap_fault+0x31/0x50 [ext4]
Nov 18 09:49:15 webapp1 kernel: [16161023.857740] [<ffffffffaedaf6a1>] ? __do_fault+0x81/0x170
Nov 18 09:49:15 webapp1 kernel: [16161023.863521] [<ffffffffaedb3e20>] ? handle_mm_fault+0xd80/0x12d0
Nov 18 09:49:15 webapp1 kernel: [16161023.869980] [<ffffffffaec24761>] ? __switch_to+0x2c1/0x6d0
Nov 18 09:49:15 webapp1 kernel: [16161023.876042] [<ffffffffaec5ee5c>] ? __do_page_fault+0x25c/0x500
Nov 18 09:49:15 webapp1 kernel: [16161023.882383] [<ffffffffaf208b58>] ? page_fault+0x28/0x30
Nov 18 09:49:15 webapp1 kernel: [16161023.888157] Mem-Info:
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] active_anon:127679 inactive_anon:1903 isolated_anon:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] active_file:725 inactive_file:1562 isolated_file:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] unevictable:0 dirty:1 writeback:0 unstable:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] slab_reclaimable:1923 slab_unreclaimable:3906
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] mapped:1350 shmem:1955 pagetables:1189 bounce:0
Nov 18 09:49:15 webapp1 kernel: [16161023.890819] free:7792 free_pcp:30 free_cma:0
Nov 18 09:49:15 webapp1 kernel: [16161023.925504] Node 0 active_anon:510716kB inactive_anon:7612kB active_file:2900kB inactive_file:6248kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:5400kB dirty:4kB writeback:0kB shmem:7820kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB pages_scanned:33805 all_unreclaimable? yes
Nov 18 09:49:15 webapp1 kernel: [16161023.955076] Node 0 DMA free:3008kB min:788kB low:984kB high:1180kB active_anon:12448kB inactive_anon:0kB active_file:0kB inactive_file:132kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB slab_reclaimable:0kB slab_unreclaimable:40kB kernel_stack:80kB pagetables:32kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
Nov 18 09:49:15 webapp1 kernel: [16161023.986635] lowmem_reserve[]: 0 555 555 555 555
Nov 18 09:49:15 webapp1 kernel: [16161023.992272] Node 0 DMA32 free:28160kB min:28208kB low:35260kB high:42312kB active_anon:498268kB inactive_anon:7612kB active_file:2900kB inactive_file:6104kB unevictable:0kB writepending:4kB present:612340kB managed:590876kB mlocked:0kB slab_reclaimable:7692kB slab_unreclaimable:15584kB kernel_stack:2432kB pagetables:4724kB bounce:0kB free_pcp:120kB local_pcp:120kB free_cma:0kB
Nov 18 09:49:15 webapp1 kernel: [16161024.026162] lowmem_reserve[]: 0 0 0 0 0
Nov 18 09:49:15 webapp1 kernel: [16161024.031581] Node 0 DMA: 8*4kB (UM) 2*8kB (U) 3*16kB (U) 1*32kB (U) 11*64kB (UM) 11*128kB (UM) 3*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3008kB
Nov 18 09:49:15 webapp1 kernel: [16161024.047958] Node 0 DMA32: 138*4kB (UME) 71*8kB (UE) 22*16kB (UE) 90*32kB (UME) 102*64kB (UM) 67*128kB (UM) 20*256kB (UM) 7*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 28160kB
Nov 18 09:49:15 webapp1 kernel: [16161024.067172] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Nov 18 09:49:15 webapp1 kernel: [16161024.076373] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Nov 18 09:49:15 webapp1 kernel: [16161024.085300] 4259 total pagecache pages
Nov 18 09:49:15 webapp1 kernel: [16161024.089482] 0 pages in swap cache
Nov 18 09:49:15 webapp1 kernel: [16161024.093181] Swap cache stats: add 0, delete 0, find 0/0
Nov 18 09:49:15 webapp1 kernel: [16161024.098961] Free swap = 0kB
Nov 18 09:49:15 webapp1 kernel: [16161024.102514] Total swap = 0kB
Nov 18 09:49:15 webapp1 kernel: [16161024.105899] 157083 pages RAM
Nov 18 09:49:15 webapp1 kernel: [16161024.109265] 0 pages HighMem/MovableOnly
Nov 18 09:49:15 webapp1 kernel: [16161024.113494] 5387 pages reserved
Nov 18 09:49:15 webapp1 kernel: [16161024.117004] 0 pages hwpoisoned
Nov 18 09:49:15 webapp1 kernel: [16161024.120448] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
Nov 18 09:49:15 webapp1 kernel: [16161024.129485] [ 416] 0 416 13954 381 24 3 0 0 systemd-journal
Nov 18 09:49:15 webapp1 kernel: [16161024.139368] [ 436] 0 436 11485 305 22 3 0 -1000 systemd-udevd
Nov 18 09:49:15 webapp1 kernel: [16161024.149222] [ 466] 0 466 1054 24 8 3 0 0 acpid
Nov 18 09:49:15 webapp1 kernel: [16161024.158493] [ 470] 0 470 9495 121 23 3 0 0 systemd-logind
Nov 18 09:49:15 webapp1 kernel: [16161024.168448] [ 471] 108 471 11283 116 26 3 0 -900 dbus-daemon
Nov 18 09:49:15 webapp1 kernel: [16161024.177974] [ 495] 0 495 7409 62 19 3 0 0 cron
Nov 18 09:49:15 webapp1 kernel: [16161024.187495] [ 604] 0 604 5120 276 12 3 0 0 dhclient
Nov 18 09:49:15 webapp1 kernel: [16161024.196753] [ 652] 0 652 3634 34 12 3 0 0 agetty
Nov 18 09:49:15 webapp1 kernel: [16161024.205847] [ 653] 0 653 3578 36 12 3 0 0 agetty
Nov 18 09:49:15 webapp1 kernel: [16161024.214941] [ 674] 106 674 24470 132 22 3 0 0 ntpd
Nov 18 09:49:15 webapp1 kernel: [16161024.224283] [ 758] 109 758 163380 14985 104 4 0 0 mysqld
Nov 18 09:49:15 webapp1 kernel: [16161024.233381] [ 817] 0 817 13692 3111 31 3 0 0 google_ip_forwa
Nov 18 09:49:15 webapp1 kernel: [16161024.243501] [ 818] 0 818 13729 3139 33 3 0 0 google_accounts
Nov 18 09:49:15 webapp1 kernel: [16161024.253358] [ 819] 0 819 13729 3158 32 3 0 0 google_clock_sk
Nov 18 09:49:15 webapp1 kernel: [16161024.263222] [ 825] 0 825 17489 213 40 3 0 -1000 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.272120] [30829] 0 30829 11208 97 27 3 0 0 sudo
Nov 18 09:49:15 webapp1 kernel: [16161024.281202] [30830] 0 30830 532390 50302 166 6 0 0 java
Nov 18 09:49:15 webapp1 kernel: [16161024.290274] [18449] 0 18449 11208 96 27 3 0 0 sudo
Nov 18 09:49:15 webapp1 kernel: [16161024.299192] [18450] 0 18450 522507 39168 136 6 0 0 java
Nov 18 09:49:15 webapp1 kernel: [16161024.308095] [ 9924] 0 9924 62529 140 29 4 0 0 rsyslogd
Nov 18 09:49:15 webapp1 kernel: [16161024.317341] [11724] 0 11724 1072 20 7 3 0 0 apt.systemd.dai
Nov 18 09:49:15 webapp1 kernel: [16161024.327190] [11728] 0 11728 1072 33 8 3 0 0 apt.systemd.dai
Nov 18 09:49:15 webapp1 kernel: [16161024.337086] [11764] 0 11764 42919 10514 88 3 0 0 unattended-upgr
Nov 18 09:49:15 webapp1 kernel: [16161024.347224] [12532] 0 12532 15889 82 32 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.356263] [12533] 0 12533 15889 82 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.365246] [12535] 0 12535 15889 82 32 4 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.374502] [12536] 0 12536 15889 82 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.383422] [12537] 0 12537 15889 81 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.392499] [12538] 0 12538 15885 74 31 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.406436] [12539] 0 12539 15353 232 30 3 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.415332] [12540] 0 12540 11888 55 24 4 0 0 sshd
Nov 18 09:49:15 webapp1 kernel: [16161024.424228] [12541] 0 12541 809 22 6 3 0 0 sshd
Nov 18 11:38:05 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="9924" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 18 11:38:05 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="13159" x-info="http://www.rsyslog.com"] start
Nov 18 22:49:45 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="13159" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 18 22:49:45 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] start
Nov 19 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 19 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 19 09:06:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="17187" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 19 09:06:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="21250" x-info="http://www.rsyslog.com"] start
Nov 19 21:01:43 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="21250" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 19 21:01:43 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24876" x-info="http://www.rsyslog.com"] start
Nov 20 06:25:02 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24876" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 20 07:44:02 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="24876" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 20 07:44:02 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="28734" x-info="http://www.rsyslog.com"] start
Nov 20 19:15:25 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="28734" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 20 19:15:25 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="32165" x-info="http://www.rsyslog.com"] start
Nov 21 06:18:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="32165" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 21 06:18:22 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="3477" x-info="http://www.rsyslog.com"] start
Nov 21 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="3477" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Nov 21 17:26:12 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="3477" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 21 17:26:12 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="6999" x-info="http://www.rsyslog.com"] start
Nov 22 02:50:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="6999" x-info="http://www.rsyslog.com"] exiting on signal 15.
Nov 22 02:50:55 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="10305" x-info="http://www.rsyslog.com"] start
Nov 22 06:25:03 webapp1 liblogging-stdlog: [origin software="rsyslogd" swVersion="8.24.0" x-pid="10305" x-info="http://www.rsyslog.com"] rsyslogd was HUPed

agraham · Nov 26, 2020

I don't pretend to understand all that but it looks a bit like an out of memory problem. There are various references to "page fault", "oom" and "out of memory" and I particularly noted "oom_kill_process". But then I could be wrong!

juvanum · Nov 26, 2020

agraham said:
I don't pretend to understand all that but it looks a bit like an out of memory problem. There are various references to "page fault", "oom" and "out of memory" and I particularly noted "oom_kill_process". But then I could be wrong!

I think you are right. thanks.
I found answers similar to yours on stackexchange.
(My server is an entry level vps.. I was hoping to see a warning from google).

Erel · Nov 27, 2020

Don't you see heap dump files in the project folder?

B4J Question b4j server down. which log file to watch

juvanum

Active Member

Erel

B4X founder

juvanum

Active Member

juvanum

Active Member

agraham

Expert

juvanum

Active Member

Erel

B4X founder

Similar Threads

B4J Question b4j server down. which log file to watch

juvanum

Active Member

Erel

B4X founder

juvanum

Active Member

juvanum

Active Member

agraham

Expert

juvanum

Active Member

Erel

B4X founder

Similar Threads

Privacy & Transparency

Privacy & Transparency