Bash script : read text files using "AND" conditions with different lines

Question

I have created simple bash script with "AND" condition but not working :

#!/bin/bash

cat log3.txt | \
while read -r LINE
do
  if [[ $LINE =~ Host ]] && [[ $LINE =~ denied ]]  ; then echo $LINE;
fi
done

and here are content of log3.txt

Host: abcd.com
Access denied

If using OR condition it is working well, but I want to use the AND condition, so if the log contains both of the strings "Host" and "Access Denied", I will get output.

Answer 1

In any single iteration of your while loop, the value $LINE can't be both Host and denied. It's impossible given the data in the file. This is why you get no output.

If you want to see all lines in the file that matches the two words Host or denied, use grep instead:

grep -wF -e 'Host' -e 'Access denied' <log3.txt

The options used here will ensure that we are doing string comparisons rather than regular expression matches (-F) and that we are matching complete words and not substrings (-w). The two query strings are given with -e and we'll get any line containing any of these.

If you want to make a slightly more advanced query, which only shows the lines that contains the two words if they both appear in the file, then you could do it with an awk program instead:

awk '/Host/ { hostline=$0 } /Access denied/ { deniedline=$0 }
     END { if ((hostline != "") && (deniedline != ""))
               print hostline; print deniedline; }' <log3.txt

Here, if we find a line matching the string Host, we save it, and likewise for the string Access denied. At the end, if both strings contain anything, we print them.

In more or less equivalent shell code:

#!/bin/sh

while IFS= read -r line; do
    case $line in
        *Host*)
           hostline=$line   ;;
        *"Access denied"*)
           deniedline=$line ;;
    esac
done <log3.txt

if [ -n "$hostline" ] && [ -n "$deniedline" ]; then
    printf '%s\n%s\n' "$hostline" "$deniedline"
fi

Here I use a case ... esac statement to do the matching on the read data. The patterns used are filename globbing patterns, not regular expressions.

Related:

• Understanding "IFS= read -r line"
• Why is using a shell loop to process text considered bad practice?

Answer 2

If you are sure that the files to process are small (like your example), you can read the whole file in one go and test that:

 file=$(<log3.txt)
 [[ $file =~ Host ]] && [[ $file =~ denied ]] && echo "$file"

For a larger file, and assuming that Host comes before denied you may use a faster (for external files) sed:

 <log3.txt sed -n '/^Host/!d;p;:1;n;/\<denied\>/{p;q};b1'

Understand that this solution will strictly print the first line that starts with Host and the following (not in the same line) first line that contains denied as a word.

If you need to extract several pairs of Host - denied, change the q to a b, that will re-start the cycle:

 <log3.txt sed -n '/^Host/!d;p;:1;n;/\<denied\>/{p;b};b1'

A similar solution with awk that will print the last Host line that is just before a denied line (in pairs):

 awk  '  p==1 && /\<denied\>/     {d=$0;p=0}
                /^Host*/         {h=$0;p=1}
         { if(p==0&&h!=""&&d!="") {print h,RS,d;p=2} }
      '  <log3.txt

And the same logic (except that it will match denied anywhere on the line (not as a word)) in the shell:

 #!/bin/sh
 p=0
 while IFS= read -r line; do
    if [ "$p" = 1 ]; then
        case $line in
           *denied*)        deniedline=$line; p=0   ;;
        esac
    fi

 case $line in
    Host*)               hostline=$line; p=1   ;;
 esac

 if [ "$p" = 0 ] && [ "$hostline" ] && [ "$deniedline" ]; then
    printf '%s\n%s\n' "$hostline" "$deniedline"
    p=2
 fi

 done <log3.txt

Answer 3

See @Kusalananda answer for why your solution doesn't work.

My solution using grep -z:

grep -zEo -e 'Host: (\w|\.)+\s+Access denied\s' log.txt

In slo-mo:

• -E: use extended regexp
• -o: print matches only
• -z: use \0 as line delimiters. Since there are none, the search is done on the whole file, where \n is just a plain character.
• - e'Host: (\w|\.)+\s+Access denied\s': look for:
  • "Host:"
  • A sequence of letter, digits, or dots
  • A space-class character (which is going to be \n)
  • "Access denied"
  • A space-class character (which is going to be \n). This one is need to get a linefeed on output

Running on:

Host: denied1.com
Access denied
Host: ok.com
Access OK
Host: random.com

Host: denied2.com
Access denied

More stuff

Yields:

Host: denied1.com
Access denied
Host: denied2.com
Access denied