1

I've a RaspberryPi 3 B running with Raspian on buster (10).

Hardware:

With that configuration, the system has worked well for several years (!) without any issues apart from 2 SD-card errors.

Some weeks ago, I've added an adaptor with realtek rtl2832u to fetch 433 MHz data. Therefore, I've installed rtl_sdr which pushes 433-Mhz weather-station data to zigbee. In general, this works well, but now, I'm facing sudden crashes of the Raspi.

Crash means:

  • Device can be ping'ed
  • no ssh connection (timeout)
  • no other server connection
  • no zigbee communicaton
  • a connection via hdmi doesn't show anything

Only solution is to unplugg power supply.

I've checked

  • dmesg
  • /var/log/kern.log
  • /var/log/messages
  • /var/log/syslog

Without finding anything. Additionally I can't find any sign of undervoltage, in particular:

pi@raspberrypi:~ $ vcgencmd get_throttled
throttled=0x0

The crash happens after some hours or a week. When unplugging the rtl2832u, everything runs smoothly again.

How shall I investigate this further?

  • Powered USB-hub? (Which devices to be connected to this, which directly to the Raspi?, USB 3.0 recommended?)
  • other power supply?
  • moving zigbee stick and rtl-stick further away (~10 cm)
  • anything else?

Update 2022-03-22

With open housing systems runs now for 26 h without any issue. Temperature is mostly between 50 and 60 deg celcius with rare spikes up to 62 deg.

Memory

pi@raspberrypi:~ $ free -m
              total        used        free      shared  buff/cache   available
Mem:            923         387          71           3         463         479
Swap:         59330         335       58995

Is there anything bad with theses numbers?

Update 2022-04-02

I've added an active usb hub to power the sdr-device. Unfortunately, the crash happend again after 3 days. :-(

Now, I'm wondering whether the issue has actually no causality with adding the stick but with re-arranging my devices (power adaptors, network switch, raspi) within a cable box. Therefore, I've removed the switch from the box (which was the case before adding the sdr-stick) as well as the sdr-stick and the active usb hub. In other words, it's almost the same setup as before.

Now, I'll see if it crashes again. If not, I'll add the sdr stick once again.

Update 2022-04-07 After 5 days of operation I see this memory usage

pi@raspberrypi:~ $ free -m
              total        used        free      shared  buff/cache   available
Mem:            923         460         115          38         347         376
Swap:            99          99           0

I don't get why total swap changes from last check. available mem slightly decreased.

I've logged temperature and cpu load in node-red: Temperature and CPU load

Now, I've plugged the sdr-stick into the pi, without starting any further software and will monitor this for several days.

Update 2022-04-08

I didn't change anything on swap.

pi@raspberrypi:~ $ cat /proc/swaps
Filename                                Type            Size            Used            Priority
/var/swap                               file            102396          102296          -2

This matches to the configuration in /etc/dphys-swapfile which contains

CONF_SWAPSIZE=100
amw
  • 173
  • 7
  • Lack of HDMI is typical if HDMI wasn't plugged in at last boot. Crashes like you describe could be caused by OOM, power brown out, or the ram chip overheating. Investigating these or improving ram heatsink might help. – user10489 Mar 20 '22 at 13:54
  • Concerning oom, `sudo grep oom /var/log/*`` I can't find any useful hint. I'll open the housing to reduce temperature. Which change would you suggest to reduce power supply issues? – amw Mar 20 '22 at 18:36
  • The USB hardware was changed on the 3B+ model; are you 3B+, or 3B? – Seamus Mar 20 '22 at 22:57
  • @user10489: OOM? – Seamus Mar 20 '22 at 23:36
  • oom: out of memory, I shoudla expanded that – user10489 Mar 21 '22 at 11:22
  • 1
    Also, checking logs after you power cycle from a crash is likely not useful. Any error (especially low power errors and dmesg) is likely lost. In particular, dmesg is an in memory buffer only. – user10489 Mar 21 '22 at 11:23
  • It's a 3B (revision a02082). – amw Mar 21 '22 at 11:58
  • Best way to reduce heat is to put a heat sinks on each of the 3 chips. If that isn't enough, a fan is next. – user10489 Mar 22 '22 at 01:05
  • these three heat sinks are already mounted on the chips. – amw Mar 23 '22 at 04:55
  • Update on 2022-03-23: After two days of operation, even with open housing and passive heat sinks, the crash happend once again. Last memory usage was quite comparable to the update in the question above.

    What would you suggest to change now? Buying a powered usb hub? Which devices shall I connect to that?

     – amw Mar 23 '22 at 16:19
  • Try adding fans first. The official power supply probably gives more power than a powered hub does. – user10489 Mar 24 '22 at 02:33
  • I wanted to use the powered usb hub not to power the raspi, but to be placed between raspi and usb devices (zigbee stick or antenna) to reduce current which has to be provided by rasp. Does that make sense? – amw Mar 24 '22 at 09:53
  • Sure. Might work. – user10489 Mar 24 '22 at 11:23
  • Unless you have some strange intermittent spiky load or intermittent ambient heat, if it was going to crash, it should have done it in less than a day. Crashing after 3-4 days makes me want to lean more towards either a memory leak (a few hundred M of cache in your numbers looks good, but maybe it shrinks after 3 days?) or your hardware is old and dying. You said this pi was a few years old, could be the memory is failing. Or the added sw to handle zigbee is using up the memory slowly. – user10489 Apr 02 '22 at 15:21
  • any chance you can just incorporate a nightly reboot? if it wasn't power, it might be software. you might be able to find the culprit via resource monitoring. – Abel Apr 02 '22 at 18:34
  • If a nightly reboot fixes it, that more or less confirms a memory leak and you can either chalk it up as solved or then hunt which app is leaking, or restart just that app. – user10489 Apr 02 '22 at 21:38
  • Thanks a lot for your ongoing support. Rebooting is not suitable as it deletes the history in my node.red dashboards. Available memory is always above 400 MiB. Wanna wait this week before pluging the stick in again. – amw Apr 04 '22 at 18:58
  • Watching the free+cached memory change daily is nearly as good as a nightly reboot anyway. – user10489 Apr 04 '22 at 23:23
  • Swap change is odd. What does cat /proc/swaps say? Maybe something is dynamically adding and removing swap? – user10489 Apr 07 '22 at 17:22
  • I have absolutely no idea. I've put the answer to the question as an update in the initial question. – amw Apr 08 '22 at 19:24

1 Answers1

0

Update 2022-04-16

After 4 days with complete setup including sdr-stick, I've seen no crash anymore. My guess is, that the crash was an EMC issue, which has been solved by moving a network switch including power supply from the raspi.

Memory as of today:

pi@raspberrypi:~ $ free -h
              total        used        free      shared  buff/cache   available
Mem:          923Mi       495Mi        70Mi        54Mi       357Mi       319Mi
Swap:          99Mi        99Mi       0.0Ki

I've seen values between 300 and 400 MiB when checking.

Thanks for all comments which helped me on that way.

amw
  • 173
  • 7