grep to find files that contain ^M (Windows carriage return)

Question

I use Linux. There is a pesky ^M (Windows cariage return) somewhere hidden in thousands of configuration files, and I have to find it, because it makes the server fail.

How do I find ^M among a directories hierarchy full of configuration files?

I think I can not enter ^M on the bash command line. But I have it in a text file that I called m.txt

"I can not enter ^M on the bash command line". Yes you can. Try control-V Control-M — Hennes, Aug 23 '16 at 11:06

score 117 · Accepted Answer · edited Sep 22 '20 at 19:18

117

grep -r $'\r' *

Use -r for recursive search and $'' for c-style escape in Bash.

Moreover, if you are sure it's a text file, then it should be safe to run

tr -d $'\r' < filename

to remove all \r in a file.

If you are using GNU sed, -i will perform an in-place edit, so you won't need to write the file back:

sed $'s/\r//' -i filename

edited Sep 22 '20 at 19:18

Community

1

answered Oct 01 '10 at 05:51

livibetter

1,897

11

@Nicolas: You can enter a ^M at the command line by pressing ^V^M, but it's better to use $'\r'. – Dennis Williamson Oct 01 '10 at 05:54
Great, it works! Thanks for the ^V^M trick too :-) – Nicolas Raoul Oct 01 '10 at 05:56
5

Under Cygwin, -U is needed to make this work. And -n will tell you the line number: grep -r -U -n -e $'\r' – Rainer Blome Jan 03 '13 at 16:16
5

Add an -l to the grep command to just view the filenames. Else you might be bombarded with matching lines. – SineSwiper Mar 25 '14 at 20:28
Note that using \r in sed 's/\r//' -i filename won't work with OS X's version of sed (I've read that GNU-sed will recognize it though). As a workaround, CTRL-V + CTRL-M seems to work in the place of \r instead. – 40XUserNotFound May 15 '14 at 21:46
¿Can $'\r' be somehow inserted inside a greater expression? – 174140 Jul 28 '15 at 06:47
@Cupcake I edited the answer, and use C-style, it should work as long as the shell is Bash. – livibetter Sep 20 '15 at 01:22
@uprego what do you mean by "greater expression?" Could you give a line of example code of what are you trying to do? – livibetter Sep 20 '15 at 01:23
@livibetter I was confused by not knowning (currently don't know yet) the meaning of the dollah glyph in sed $'s/\r//' -i filename, use a dollar before a substitution expresion. Never seen it. And I was confused by the uses in grep -r $'\r' * and tr -d $'\r' < filename, I was matching them to the idiom join file1 file12 -t $'\t', like used in https://stackoverflow.com/questions/1722353. – 174140 Sep 21 '15 at 18:48
1

@uprego not sure if you understand them now, but fyi and other's, search $' read the first hit in manpage bash(1), basically, you can see it as if you were writing C literal string. As for the command < filename, the use of < or > is called redirection, this is first time I have seen anyone called it greater expression. Search REDIRECTION in bash(1). – livibetter Sep 23 '15 at 02:03
For those who happen to be using SunOS 5.10 and met with the error grep: illegal option -r , you should use grep $'\r' * – Nasri Najib Oct 25 '17 at 11:51
how to exclude searching for \r\n? – machinarium Jul 17 '19 at 14:48

score 15 · Answer 2 · edited Jul 25 '12 at 23:25

15

When I tried, I could tell it was sort-of working, but the lines were printing blank. Add in the option:

--color=never

If you get this issue, I think it's the escape characters for color highlighting interfering with the \r character.

edited Jul 25 '12 at 23:25

Gareth

18,809

answered Jul 25 '12 at 22:58

Judson Wilson

151

Thanks! I did not know why I was getting blank lines. – KolaB Sep 07 '20 at 16:07

mivk · Answer 3 · 2020-09-07T12:13:01.453

3

If I understand your question correctly, what you really want is to normalize all line-endings to the Unix LF (\x0a) standard. That is not the same as just blindly removing CRs (\x0d).

If you happen to have some Mac files around which use just CR for newlines, you will destroy those files. (Yes, Macs are supposed to use LF since almost 20 years, but there are still (in 2019) many Mac apps which use just CR).

You could use Perl's \R linebreak escape to replace any sort of newline with \n.

perl -i.bak -pe 's/\R/\n/g' $your_file

This would replace in-place any sort of linebreak with \n in $your_file, keeping a backup of the original file in ${your_file}.bak.

edited Sep 07 '20 at 12:13

answered Aug 16 '19 at 19:59

mivk

3,726

dos2unix is the proper command. – pbies Apr 28 '22 at 09:16
@pbies: Unfortunately, no. dos2unix doesn't seem to work in cases of mixed line endings (just re-tested on files with mixed CRLF and CR-only). That was the original reason why I started to use Perl instead. Besides, dos2unix needs to be installed, while Perl is already available on any decent Unix system. – mivk Apr 29 '22 at 15:12
Have you tried binary mode? – pbies Apr 29 '22 at 19:11

score 3 · Answer 4 · edited Jul 09 '15 at 05:55

If your server does not have a bash shell, an alternative is to use the -f option on grep, in combination with a prepared file containing \r.

To create the file:

$ echo -ne '\r' > /tmp/cr                    --or--                   $ printf '\r' > /tmp/cr

$ od -c /tmp/cr
0000000  \r
0000001

To actually do the search

$ grep -f /tmp/cr *.html *.php *.asp *.whatever

or you can be a little lazy and just type *,

$ grep -f /tmp/cr *

The -f filename option on grep is used to specify a file that contains patterns to match, one per line. In this case there's only one pattern.

score 1 · Answer 5 · edited Nov 20 '20 at 02:40

1

To use grep on end-of-line characters, I guess you have to tell grep the file is binary.

-l (letter L) is for printing only the filename
-P is for perl regexp (so \x0d is transformed to \r or ^M)

grep -l --binary -P '\x0d' *

edited Nov 20 '20 at 02:40

phuclv

27,773

answered Aug 23 '16 at 11:00

Vouze

216

score 0 · Answer 6 · answered Apr 28 '22 at 09:00

Other answers require Bash, this one should not:

grep -a -r "$(printf '\r')"

Explanation

printf '\r' prints a literal carriage return character
The wrapping "$(..)" puts the CR into an argument to the grep command.
-a tells grep to act on binary files, too, so that it actually prints matching lines even if the file is considered binary.

score 0 · Answer 7 · edited Jan 19 '12 at 09:23

0

In regular expression style, various newlines:

Windows (CR LF)
\r\n

Unix (LF)
\n

Since the \r\n sequence is fairly unique, I think you should be able to search for it that way?

To make things worse Macs used to have just '\r' in place of newline. I cannot verify this, but I don't think MacOSX generations does that any more.

Older Macs (CR)
\r

edited Jan 19 '12 at 09:23

Manwe

888

answered Oct 01 '10 at 05:48

Jeff Atwood

24,142

In the directory that contains m.txt, grep "\r\n" * gives no result. No result either for egrep -e "\r\n" * nor grep -E "\r\n" * – Nicolas Raoul Oct 01 '10 at 05:54
@nicolas ah, I misunderstood.. you meant CR only \r my bad. A full windows newline is indeed \r\n or CRLF – Jeff Atwood Oct 01 '10 at 05:58

score 0 · Answer 8 · answered Oct 10 '11 at 05:11

0

If you are on a Mac and use homebrew, you can do:

brew install tofrodos
fromdos file.txt

to remove all the Windows carriage returns from file.txt

To switch back to Windows carriage returns,

todos file.txt

answered Oct 10 '11 at 05:11

kortina

101

to search in a folder and clean all files coming from dos, run this command : find . -type f -name "*.java" | xargs fromdos – Taiko May 19 '14 at 03:38

score 0 · Answer 9 · edited Nov 20 '20 at 02:39

0

Following up on previous answers, the tr method is good:

533$ if [[ -n "`tr -cd "\r" <~/.bashrc`" ]]; then echo "DOS"; else echo "UNIX"; fi
UNIX
534$ if [[ -n "tr -cd &quot;\r&quot; &lt;dosfile.txt" ]]; then echo "DOS"; else echo "UNIX"; fi
DOS

edited Nov 20 '20 at 02:39

phuclv

27,773

answered Nov 21 '14 at 06:23

Malcolm Boekhoff

97

grep to find files that contain ^M (Windows carriage return)

9 Answers9

Explanation