1

Consider these two commands run against my cluster.

First roll up all known used ram as reported by top nodes:

$ kubectl --no-headers=true top nodes | awk 'BEGIN { total=0 } { total+=$4 } END { print total, "Mi" }'
504972 Mi

Then by all pods in top pods

phs@anvil:~$ kubectl --no-headers=true --all-namespaces=true top pods |  awk 'BEGIN { total=0 } { total+=$4 } END { print total, "Mi" }'
163503 Mi

I've already confirmed that all living pods appear in top pods.

What is with the huge discrepancy? Where does the extra ram go?

phs
  • 613

1 Answers1

3

It appears the difference comes down to buffers. If we run free -mw on each node I can see significant (10-15 GB) amounts of ram sunk there.

$ kubectl get nodes -o name | cut -d / -f 2 | while read node; do ssh -n admin@$node 'free -mw | fgrep Mem:'; done | awk 'BEGIN { total=0 } { total+=$6 } END { print total, "Mi" }'
278815 Mi

Their total is approximately the missing mount (these commands were run at different times, and so exact equality shouldn't be expected.)

phs
  • 613