Writing to /dev/kmsg randomly shows up in /proc/kmsg and always in dmesg?

Question

When writing a message to /dev/kmsg while tailing /proc/kmsg the messages show up very randomly, in never shows up in /var/log/kern.log.

However, dmesg gets all the messages correctly.

Example

# Terminal 1
sudo tail -f /proc/kmsg

# Terminal 2 
tail -f /var/log/kern.log

# Terminal 3
sudo sh -c 'echo "<1> Kernel log message" > /dev/kmsg'
sudo sh -c 'echo "<1> Kernel log message" > /dev/kmsg'
sudo sh -c 'echo "<1> Kernel log message" > /dev/kmsg'
sudo sh -c 'echo "<1> Kernel log message" > /dev/kmsg'

Results:

# Terminal 1
# The message is shown at irregular intervals without a clear pattern
sudo tail -f /proc/kmsg
    <9>[37102.185491]  Kernel log message

# Terminal 2 
# No message is ever shown
tail -f /var/log/kern.log

# Terminal 4
# Everything is always here
dmesg
    [37101.556366]  Kernel log message
    [37102.185491]  Kernel log message
    [37103.042422]  Kernel log message

When plugging in an USB stick, the data gets randomly split between output from /proc/kmsg and /var/log/kern.log. dmesg again show all the messages.

• What is dmesg doing differently from other approaches?
• Why is it possible to just break rsyslog's kernel logging by having another process listen to /dev/kmsg. Why aren't my attempts blocked with device or resource busy?
• Why doesn't rsyslog ever log the messages sent to /dev/kmsg even though they do occasionally reach proc/kmsg?

Answer 1

/proc/kmsg is not shareable amongst log readers, and if there are multiple readers only one of them will receive any given log message. /dev/kmsg is shareable amongst multiple log readers, providing the whole log stream (pace underruns by slow readers) to each reader.

Further reading

• https://unix.stackexchange.com/a/294206/5132