2

our goal is to create bash script that delete the unused / unnecessary UUID number/s from /etc/fstab file ,

brief background - in our labs , we have more then 500 RHEL servers , and we want to fix the fstab files that have incorrect fstab configuration as unused UUID number/s or unused UUID number/s that are in Comment lines , etc

we create the following bash script as example.

#!/bin/bash

blkid_list_of_uuid=blkid | awk -F'UUID=' '{print $2}' | awk '{print $1}' | sed s'/"/ /g'
grep UUID /etc/fstab >/tmp/fstab


while read line_from_fstab
do
        echo "checking if ${line_from_fstab} is unused UUID"
        if [[ ! ${line_from_fstab} =~ $blkid_list_of_uuid ]]
        then
           #sed -i "/$line_from_fstab/d" /etc/fstab
           echo "delete unused line ${line_from_fstab} from fstab"
        fi
done < /tmp/fstab

we captured the blkid number in blkid_list_of_uuid variable. and filter the UUID lines from fstab into /tmp/fstab file

the target of the if syntax - [[ ! ${line_from_fstab} =~ $blkid_list_of_uuid ]] is to delete by sed ( for now in comment ) the UUID lines in /etc/fstab that are not in used

but the regex isn't working, and script actually delete the UUID that are in used

example of blkid

blkid
/dev/mapper/vg-VOL_root: UUID="49232c87-6c49-411d-b744-c6c847cfd8ec" TYPE="xfs"
/dev/sda2: UUID="Y5MbyB-C5NN-hcPA-wd9R-jmdI-02ML-W9qIiu" TYPE="LVM2_member"
/dev/sda1: UUID="0d5c6164-bb9b-43f4-aa9b-092069801a1b" TYPE="xfs"
/dev/mapper/vg-VOL_swap: UUID="81140364-4b8e-412c-b909-ef0432162a45" TYPE="swap"
/dev/mapper/vg-VOL_var: UUID="e1574eeb-5a78-4a52-b7e3-c53e2b8a4220" TYPE="xfs"
/dev/sdb: UUID="547977e2-a899-4a75-a31c-e362195c264c" TYPE="ext4"
/dev/mapper/vg-VOL_docker: UUID="2e1a2cbf-9920-4e54-8b6b-86d0482c5f7b" TYPE="xfs"
/dev/sdc: UUID="1a289232-0cfe-4df7-9ad5-6a6e2362a1c5" TYPE="ext4"
/dev/sdd: UUID="91493d1f-ffe9-4f5f-aa6d-586d2c99f029" TYPE="ext4"
/dev/sde: UUID="f11845e7-1dcb-4b81-a1d4-9a5fe7da6240" TYPE="ext4"
yael
  • 13,106
  • This is the kind of thing where I would probably reach for a programming language other than Bash. – shadowtalker Oct 27 '23 at 12:02
  • 3
    This is the kind of thing where you wonder why you have 500 servers, but individually managed /etc/fstabs with manually added comments and obsolete entries? How did that happen? Which administrative steps are you taking to let that never happen again? – Marcus Müller Oct 27 '23 at 12:07
  • [edit] your question to include concise, testable sample input (output of blkid and contents of /tmp/fstab before the script runs) and expected output (contents of /tmp/fstab after the script runs) so we can help you come up with the best solution. With what you have in your question right now we'd just be guessing at what that might be. – Ed Morton Oct 27 '23 at 12:47
  • 1
    we need your /tmp/fstab. – Marcus Müller Oct 27 '23 at 15:01
  • @yael sample blkid is 1/3 of what we need to give you the best answer and be able to test a potential solution, see my previous comment for what else we need (but I see now it's /etc/fstab rather than /tmp/fstab). – Ed Morton Oct 27 '23 at 15:22
  • We've got hundreds of servers, each with it's own /etc/fstab. They're all in their own little walled off subnet gardens, for each customer, who pays for everything themselves. OP might be in a similar situation. – RonJohn Oct 27 '23 at 20:57
  • 1
    If you don't have any special use case, why not generate new fstab from scratch, e.g. using genfstab -U /? It would be a lot simpler than trying to parse, modify and retain the original file without breaking anything. (Backup/Rename the original files. Check a few at random or where the size difference is the largest.) – frostschutz Oct 28 '23 at 09:00
  • @RonJohn hm, but that would be exactly the situation where I'd become very organized about centralized configuration management. What happens when one of the customers' servers goes down? How are you going to restore it? Manually install an OS, then do the exact same manual fiddling again hoping to remember that, and why you did it? – Marcus Müller Oct 30 '23 at 16:13
  • @MarcusMüller if a customer's server goes down, we can restore from the last snapshot. (They're *all* -- well, 99% -- on ESX VMs.) – RonJohn Oct 30 '23 at 16:19
  • @RonJohn ah, I understood "server" to be a physical machine. – Marcus Müller Oct 30 '23 at 16:21

4 Answers4

3

The reason it isn't working is because you are trying to match the wrong things. This is what your blkid variable contains:

$ printf '%s\n' "$blkid_list_of_uuid"
 49232c87-6c49-411d-b744-c6c847cfd8ec 
 Y5MbyB-C5NN-hcPA-wd9R-jmdI-02ML-W9qIiu 
 0d5c6164-bb9b-43f4-aa9b-092069801a1b 
 81140364-4b8e-412c-b909-ef0432162a45 
 e1574eeb-5a78-4a52-b7e3-c53e2b8a4220 
 547977e2-a899-4a75-a31c-e362195c264c 
 2e1a2cbf-9920-4e54-8b6b-86d0482c5f7b 
 1a289232-0cfe-4df7-9ad5-6a6e2362a1c5 
 91493d1f-ffe9-4f5f-aa6d-586d2c99f029 
 f11845e7-1dcb-4b81-a1d4-9a5fe7da6240

This means that this:

if [[ ! ${line_from_fstab} =~ $blkid_list_of_uuid ]]

becomes something like this:

if [[ ! "UUID=0a3407de-014b-458b-b5c1-848e92a327a3 /     ext4   defaults  0      1" =~ " 49232c87-6c49-411d-b744-c6c847cfd8ec 
 Y5MbyB-C5NN-hcPA-wd9R-jmdI-02ML-W9qIiu 
 0d5c6164-bb9b-43f4-aa9b-092069801a1b 
 81140364-4b8e-412c-b909-ef0432162a45 
 e1574eeb-5a78-4a52-b7e3-c53e2b8a4220 
 547977e2-a899-4a75-a31c-e362195c264c 
 2e1a2cbf-9920-4e54-8b6b-86d0482c5f7b 
 1a289232-0cfe-4df7-9ad5-6a6e2362a1c5 
 91493d1f-ffe9-4f5f-aa6d-586d2c99f029 
 f11845e7-1dcb-4b81-a1d4-9a5fe7da6240 
" ]]

Of course, this will never be true: you are searching for the entire fstab line in the list of found UUIDs. What you wanted to do is search for the UUID only.

Don't do this, use one of the methods given in the other answers, using the shell for this sort of thing is a bad idea, but for the sake of completion, here's a mostly shell based approach using the logic I think you meant to use (note that this requires GNU grep):

$ grep -oP '^UUID=\S+' /etc/fstab | sed 's/=/="/; s/$/"/' | 
  while read -r fstab; do 
    sudo blkid |   
      grep -q "$fstab" && 
        echo "GOOD: $fstab" || 
          echo "BAD: $fstab"; done
BAD: UUID="e16a3de8-a58f-430f-b80f-3d87e9fb0b1d"
BAD: UUID="ef6747e2-f802-4b18-9169-ae65f9933ef1"
BAD: UUID="b00792c8-f7e0-4448-b98d-021eede31e6c"
GOOD: UUID="32133dd7-9a48-4b9d-b2e0-6e383e95631d"
GOOD: UUID="69ae5a79-9a15-489c-951d-1e0c2a16b7fc"
GOOD: UUID="6E5E-90F0"
GOOD: UUID="ff3c9de1-417c-4c4d-8150-a89d222ae60b"

The BAD: are the UUIDs in my /etc/fstab file that are not found in the output of blkid on my system.

terdon
  • 242,166
2

Without sample input and expected output it's an untested guess but I think this is what you want, using any awk:

#!/usr/bin/env bash

tmp=$(mktemp) || exit 1
trap 'rm -f "$tmp"; exit' EXIT


blkid |
awk '
    NR == FNR {
        gsub(/"/,"",$2)
        blkids[$2]
        next
    }
    !/^UUID/ || ($1 in blkids)
' - /etc/fstab > "$tmp" &&
mv -- "$tmp" /etc/fstab

The above is assuming blkid output format as shown at https://linux.101hacks.com/unix/blkid:

/dev/sdb1: UUID="6e0acfe3-81ed-4f9f-8ab5-0d65ba1f0ef2" TYPE="ext2"
/dev/sdc1: UUID="aa82d7bb-ab2b-4739-935f-fd8a5c9a6cb0" TYPE="ext2"
/dev/sda1: UUID="187171ab-c9b8-43ec-b0bb-77c736ca22e0" TYPE="ext4" LABEL="/home"
/dev/sda2: UUID="1a225baa-7027-4619-aaa5-900e24c1fdff" TYPE="swap"
/dev/sdb3: UUID="2a294b33-eb61-40a3-b3fc-ad6eaf7f156f" TYPE="ext2"

and /etc/fstab contents format as shown at https://wiki.archlinux.org/title/fstab:

# <device>                                <dir> <type> <options> <dump> <fsck>
UUID=0a3407de-014b-458b-b5c1-848e92a327a3 /     ext4   defaults  0      1
UUID=f9fe0b69-a280-415d-a03a-a32752370dee none  swap   defaults  0      0
UUID=b411dc99-f0a0-4c87-9e05-184977be8539 /home ext4   defaults  0      2

as found by a quick google search.

Ed Morton
  • 31,617
  • 1
    Do you know for a fact that no systems can have a device path that contains whitespace? It seems like a safe assumption, but I don't know. It might be worth using -F: to avoid the issue. – terdon Oct 27 '23 at 13:48
  • @terdon - fwiw the default output of blkid is designed for human consumption... e.g. blkid -s UUID -o export would be much safer/easier to process ; or use lsblk which has even better control on output formatting – don_crissti Oct 27 '23 at 14:23
  • Oooh, thanks, @don_crissti, blkid -s UUID -o export is indeed a better format. – terdon Oct 27 '23 at 14:25
  • 1
    @terdon if you're asking me, I've never heard of blkid so I'm just guessing at what it might output based on what google returned (which apparently matches the example the OP has now added to their question). If the OP posts sample input/output that my answer doesn't work for it'll be an easy tweak to massage my answer to suit. – Ed Morton Oct 27 '23 at 15:21
  • I was asking (although "idly wondering" might be a better term) if dev paths (e.g. /dev/path to /sda1) can contain whitespace. I am reasonably sure they can't, I just don't know it for sure. – terdon Oct 27 '23 at 15:29
1

I am not sure you can use a list of uuid as a regexp.

to get this list I used

blkid | awk -F\" '{for(i=1;i<=NF;i++) if ($i ~ / UUID=/ ) { i++; print $i ; break ; }  }'

where

  • -F\" use " as separator
  • for(i=1;i<=NF;i++) walk the line
  • if ($i ~ / UUID=/ ) if UUID= is found (space before UUID is mandatory) ...
  • { i++ ; print $i ; break ; } }' .. get next field (actual UUID) and print it, exit loop.

next I use a plain grep

if ! echo "${blkid_list_of_uuid}" | grep ${line_from_fstab}

(you can use grep -q to lower garbage)

this result in

blkid_list_of_uuid=$(blkid | awk -F\" '{for(i=1;i<=NF;i++) if ($i ~ / UUID=/ ) { i++ ; print $i ; break ; } }' )
grep UUID /etc/fstab >/tmp/fstab
while read line_from_fstab
do
        echo "checking if ${line_from_fstab} is unused UUID"
        if ! echo "${blkid_list_of_uuid}" | grep ${line_from_fstab}
        then
           #sed -i "/$line_from_fstab/d" /etc/fstab
           echo "delete unused line ${line_from_fstab} from fstab"
        fi
done < /tmp/fstab

  • there is rarely a need to pipe two awk and a sed

  • blkid_list_of_uuid can be store in a temporary file too resulting in 

     grep -q ${line_from_fstab} /tmp/blkid_list_of_uuid

    (won't work as ${line_from_fstab} is not an uuid)

  • as pointed out, deploying and running bash script in 500 VM might not be a good idea on long run, I would consider ansible.

Archemar
  • 31,554
  • 1
    Why wouldn't you be able to use the UUID as a regex? Are you thinking it might contain special characters like . or * etc? If not, it's just a string and a string is, by definition, a regular expression matching itself. – terdon Oct 27 '23 at 13:50
0

terdon answered what you did wrong. I'll just provide an example of a simpler way to do what you tried:

lsblk -no uuid | grep . | grep -Ff- /etc/fstab

That should output fstab how you wanted it, with the difference that this isn't treating the UUIDs like regexes or potentially sed commands. You can direct the output to /tmp/fstab then move it to /etc/fstab.

I used lsblk instead of blkid because the manpage to blkid recommends lsblk instead, and lsblk doesn't require root priviledges while blkid apparently does.

JoL
  • 4,735
  • Why not use this procedure to simply list invalid fstab's - if you autocorrect one erroneously, you will only discover this during a reboot of the server. Also, do take a log of all the changes you make. – Jeremy Boden Oct 28 '23 at 00:39
  • @JeremyBoden Because that's not what the OP is doing. Autocorrection is what they're trying to do. They're trying to remove each line that doesn't match the existing UUIDs. This answer doesn't preclude them reviewing the results before overwriting. Logging may be something they want to do or it may not. It's tangential to the question, and there's other options, like recovering the fstabs from a backup if a reboot fails. – JoL Oct 28 '23 at 00:51