B+ Pi hangs what watchdog settings do I need?

Question

This has been going on for a while.

Raspbian B+ regularly updated and kernel updated, nothing connected other than Ethernet cable and power supply is Pi rated(from ModMyPi with a ModMyPi noodle cable).

This Pi has UV4L and Motion installed and . With a NOIR camera set to capture images when threshold of change met. Images are moved off the Pi and there is plenty of space.

Basically this setup works and will normally run for 2-3 days before no more images are captured. At this point I can ping the Pi but not ssh in. No errors, just never completes after entering the password.

I can not find any issues in the logs and after finding a daily cron based reboot was not enough to resolve the issue I have installed the watchdog daemon. Unfamiliar with the watchdog settings I have set the one from most tutorials max-load-1 = 24

But it still seems to get stuck.

I tried ssh-ing in after it started up and left top running when that failed I captured the following.

top - 07:36:02 up 2 days, 10:38,  2 users,  load average: 1.85, 1.80, 
Tasks: 1555 total,   1 running, 1554 sleeping,   0 stopped,   0 zombie
%Cpu(s): 88.3 us, 11.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si
KiB Mem:    380780 total,   370264 used,    10516 free,      180 buffe
KiB Swap:   102396 total,   101344 used,     1052 free,    19092 cache

COMMAND             PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM 
motion             2211 root      20   0  104m  44m  13m S  65.7 12.0 
top                2514 pi        20   0  7452 2752 1468 R  30.7  0.7 
uv4l               2201 root      rt   0  118m  22m  13m S   2.1  6.0 
rcu_preempt           7 root      20   0     0    0    0 S   0.6  0.0 
kworker/0:1          19 root      20   0     0    0    0 S   0.2  0.0 
VCHIQ-0              30 root       1 -19     0    0    0 S   0.2  0.0 
ksoftirqd/0           3 root      20   0     0    0    0 S   0.1  0.0 
ifplugd            1569 root      20   0  1752 1124 1104 S   0.1  0.3 
ifplugd            1626 root      20   0  1752 1056 1036 S   0.1  0.3 
kworker/u2:1      15529 root      20   0     0    0    0 S   0.1  0.0 
init                  1 root      20   0  2148 1044 1024 S   0.0  0.3 
kthreadd              2 root      20   0     0    0    0 S   0.0  0.0 
kworker/0:0H          5 root       0 -20     0    0    0 S   0.0  0.0 
rcu_sched             8 root      20   0     0    0    0 S   0.0  0.0 
rcu_bh                9 root      20   0     0    0    0 S   0.0  0.0 
khelper              10 root       0 -20     0    0    0 S   0.0  0.0 
kdevtmpfs            11 root      20   0     0    0    0 S   0.0  0.0 
netns                12 root       0 -20     0    0    0 S   0.0  0.0 
perf                 13 root       0 -20     0    0    0 S   0.0  0.0 
khungtaskd           14 root      20   0     0    0    0 S   0.0  0.0 
writeback            15 root       0 -20     0    0    0 S   0.0  0.0 
crypto               16 root       0 -20     0    0    0 S   0.0  0.0 
bioset               17 root       0 -20     0    0    0 S   0.0  0.0 
kblockd              18 root       0 -20     0    0    0 S   0.0  0.0 
rpciod               20 root       0 -20     0    0    0 S   0.0  0.0 
kswapd0              21 root      20   0     0    0    0 S   0.0  0.0 
fsnotify_mark        22 root      20   0     0    0    0 S   0.0  0.0 
nfsiod               23 root       0 -20     0    0    0 S   0.0  0.0 
kthrotld             29 root       0 -20     0    0    0 S   0.0  0.0 
VCHIQr-0             31 root       1 -19     0    0    0 S   0.0  0.0 
VCHIQs-0             32 root       0 -20     0    0    0 S   0.0  0.0 
iscsi_eh             33 root       0 -20     0    0    0 S   0.0  0.0 
dwc_otg              34 root       0 -20     0    0    0 S   0.0  0.0 
DWC Notificatio      35 root       0 -20     0    0    0 S   0.0  0.0 
VCHIQka-0            37 root      20   0     0    0    0 S   0.0  0.0 
SMIO                 38 root      10 -10     0    0    0 S   0.0  0.0 
deferwq              39 root       0 -20     0    0    0 S   0.0  0.0 
mmcqd/0              41 root      20   0     0    0    0 S   0.0  0.0 

Is there any indication what is wrong and what I can ask the watchdog to watch?

I also found the message watchdog stopped in the dmesg even though running /etc/init.d/watchdog status returns [ ok ] watchdog is running.

I ran tail -f /etc/var/syslog in an ssh session and tried ssh pi@192.168.20.15 uptime periodically. When that failed I ^C the tail -f and was immediately given the shell prompt but nothing I typed showed up or executed.

Answer 1

Your RAM and swap usage in the upper bar of top look like pretty much everything is used up:

KiB Mem:    380780 total,   370264 used,    10516 free,      180 buffe
KiB Swap:   102396 total,   101344 used,     1052 free,    19092 cache

The cache is almost non-existent. This will hurt the performance of the system. The kernel uses free RAM as a file/page cache. The amount available for this is actually not equivalent to what's reported above as "free" -- that figure excludes the cache we're talking about, so it would be:

total - used + cache

Here's a B+ I have running right now:

KiB Mem:    448376 total,   400940 used,    47436 free,    31288 buffers
KiB Swap:        0 total,        0 used,        0 free,   327488 cached

The "Swap" stuff is 0 because I don't have any, and the last number on that line isn't actually about swap, it's the cache size, which is in RAM. So apparently I have ~401 MiB used and only 47.4 free, but if you subtract the cache, you get 401 - 327 = 74 MiB actually used by processes. So the kernel sees 448 - 74 = 374 MiB free, and it has gradually filled that with cache. In other words, the cache is 5x bigger than the RAM that is committed to processes!

In your case, you really have used ~370 MiB of RAM, leaving very little for the cache, which is ~19 MiB, 20x smaller. The cache is used to hold previously accessed things when they are no longer officially in use by any process, in case some process wants to access them again in the near future. This way they do not have to be constantly read and written to/from disk. This is why it's a good idea to try and leave some RAM free; it will still all appear used, but part of it will be that cache.

Again, not having enough cache will hurt performance, but that's not why your system locked up at that point.

 Apr 10 08:55:58 pideskcam kernel: [129513.787567] top invoked oom-killer: 
                                                   ^^^^^^^^^^^^^^^^^^^^^^

The OOM (out of memory) killer is a kernel mechanism (here's a more in-depth explanation) that deals with running out of memory. This is not as simple as saying, "You can't do this now because there isn't enough memory". It will still let anybody do anything, but what will then immediately happen is something will ask for memory the system cannot provide, and then something has to go.

The heuristic for determining what will go is described in the "more detailed explanation". It is not necessarily what's "invoked oom-killer" (in this case, top); if you look further down in syslog after that you'll eventually find Out of memory: Kill process 1234 (some process name). In this case, it was motion. In between those points you'll find a long table explaining an evaluation (it's scoring all running process, and the one with the worst score loses).

That would still impede top, and depending on what the real culprit is (it is not the straw that broke the camel's back, which is why it chose motion and not top), the OOM killer may have to try several things before it solves the problem. This could include killing other processes arbitrarily such as sshd; if your system has rolled over logs (e.g. /var/log/syslog.1 or /var/log/syslog.gz.1) there might be a record in there.

If you press f with top running you can change the sort criteria; go down to %MEM, press s then Esc to go back and you'll see stuff displayed in order of memory usage. If you watch this for a while you should be able to determine what's eventually growing out of control.