Auto-reboot on hang

Question

I'm using my Raspberry Pi for data logging. I have a cron job that collects data every five minutes. The Pi is connected to the internet to another computer that collects this data. It sometimes happens that the Pi hangs. I can't SSH to it and no data is collected until I reboot it. This is problematic because I can't constantly monitor if everything is working. So I'd like to know if it's possible that the Pi would detect these hangs and would reboot itself.

This is the syslog that shows the Pi hanged at 14:35 on 6th March (the last cron job was executed then): http://pastebin.com/niV7LKZ3 It did not log anything before I manually rebooted it.

Yes, because it would still lead to some loss of data. Even If it would reboot every hour there's a possibility that I will lose 11 data points. — shrx, Mar 08 '14 at 09:43
Also, the Pi is used as a web server and I can't afford it to reboot every five minutes. — shrx, Mar 08 '14 at 09:52
I'm thinking this is rather impossible. If you can't SSH in and since no data is being collected, the Pi is probably entirely frozen. If this is the case than I think it's very doubtful that there is a simple solution to this issue, as it would require low-level access to the system and real-time system status monitoring at all times. One possible solution, if you can confirm that the Pi isn't entirely hung, is to have a simple python script started on boot that checks the syslog data every so often and if nothing has happened within the last 10 minutes, reboot. — RPiAwesomeness, Mar 09 '14 at 01:09

score 1 · Answer 1 · answered Mar 08 '14 at 01:02

1

The traditional solution to this problem is a watchdog timer. This requires a small amount of external hardware.

answered Mar 08 '14 at 01:02

Milliways

59,890
31
101
209

1

I found this program named watchdog. What external hardware do you have in mind? – shrx Mar 08 '14 at 09:44
The usual is a timer which triggers the reset. Typically this would be monostable with 10 or 30 sec timeout. You include code in your program which resets this each time around the loop, so that if the program locks up it doesn't reset the timer. – Milliways Mar 08 '14 at 12:02
1

The Pi actually has a watchdog on it's own that might be sufficient. The kernel module needed is called bcm2708_wdog. – Izzy Jan 13 '15 at 22:43

score 1 · Answer 2 · answered Mar 08 '14 at 07:10

Two questions: A- Which build are you running? B- How long does the information gathering period take?

If it is almost instantaneously this is an easy fix and the following should work for you:

1- Create a new bash script and name it anything you want (extension .sh)

sudo touch yourFile.sh

2- Edit the permissions of this file to make it executable to everyone

sudo chmod a+x yourFile.sh

3- Edit your .bashrc file with the following command

sudo nano .bashrc

4- Add the following line to the bottom of .bashrc so your file will run on boot

./yourFile.sh

5- You can also add the same to your /etc/profile file - again at the bottom

. /home/pi/yourFile.sh

6- Edit your /etc/inittab file to enable autologin with the following command

sudo nano /etc/inittab

7- About 80% down the page you will need to make the following changes

Comment these two lines out:

1:2345:respawn:/sbin/getty 38400 tty1
2:23:respawn:/sbin/getty 38400 tty2

Add these two lines below them:

#1:2345:respawn:/sbin/getty 38400 tty1
#2:23:respawn:/sbin/getty 38400 tty2
1:2345:respawn:/bin/login -f pi tty1 </dev/tty1 >/dev/tty1 2>&1
2:23:respawn:/bin/login -f pi tty2 </dev/tty2 >/dev/tty2 2>&1

8- Lastly, put the following inside the yourFile.sh file you created

#!/bin/bash  

currentTime=`date +%M`
checkTime()
{
    nowTime=$1
    if [ $(( $nowTime % 5 )) -eq 0 ] ; then
            echo `sleep 180`
            echo `sudo reboot`
    elif [ $(( $nowTime % 5 )) -eq 1 ] ; then
            echo `sleep 120`
            echo `sudo reboot`
    elif [ $(( $nowTime % 5 )) -eq 2 ] ; then
            echo `sleep 60`
            echo `sudo reboot`
    elif [ $(( $nowTime % 5 )) -eq 3 ] ; then
            echo `sudo reboot`
    elif [ $(( $nowTime % 5 )) -eq 4 ] ; then
            echo `sleep 240`
            echo `sudo reboot`
    fi
}
checkTime $nowTime

The pi will reboot anywhere from XX:X3 and XX:X4 (1 to 2 minutes before your cron job is scheduled to run. The normal boot time takes about 30-40 seconds so no worries there.

You can definitely tweak the number to suit your needs.

Let me know how it works!!

On one Pi I have hardFP raspbian, on the other I have softFP debian wheezy. I'd like a method that works with both. The data collection usually takes a couple of seconds and never more than a minute. — shrx, Mar 08 '14 at 09:47
I'm afraid I can't use this method, because the Pi is used as a web server and I can't afford it to reboot every five minutes. — shrx, Mar 08 '14 at 09:53
This may not fix the issue, as if the Pi is completely frozen it won't run the code you have written, and thus the above answer is more correct. — Ian M, Mar 02 '15 at 10:52

Auto-reboot on hang

2 Answers2