53

What is the sed incantation to remove null bytes from a file? I'm trying:

s/\000//g

but that is stripping out strings of zeroes.

s/\x00//g

seems to have no effect. I'm trying to do this in a sed script, so I'm not sure the echo trick will work.

the
  • 2,819
Chris Curvey
  • 1,427

4 Answers4

64

I don't know how you can exactly achieve this with sed, but this is a solution that works with tr:

tr < file-with-nulls -d '\000' > file-without-nulls

This is a solution for sed that works with some (but not all) implementations of sed:

sed 's/\x0//g' file1 > file2

This is a solution that involves replacing into space characters which should work in all occasions:

sed 's/\x0/ /g' file1 > file2
Codebling
  • 734
Tamara Wijsman
  • 57,463
  • 10
    that looks like a very incomplete answer. why would it work on some occasions and not others, and if so then wouldn't an example be useful? – barlop May 24 '11 at 19:11
  • @barlop: Because of the way it is implemented? The OP didn't specify one and I'm not going to enumerate every single implementation... – Tamara Wijsman May 24 '11 at 20:50
  • 4
    Well that then sounds fine to me, so you're saying it depends on the implementation of SED. If you hadn't said that you'd have left open the possible suggestion that one implementation of SED may remove nulls from one file and not another file, depending on the data in the file. – barlop May 24 '11 at 22:03
  • 3
    shouldn't it be "tr -d '\000' < file-with-nulls > file-without-nulls" ? – Seamus Abshere Jul 12 '13 at 19:32
  • I needed to convert nul to newline, sed 's/\x0/\n/g' worked for me – doug65536 Feb 07 '14 at 07:33
  • 2
    Works for me™. Also useful: -i parameter to convert file in place. – zbyszek Apr 19 '16 at 19:20
  • tr < file-with-nulls -d '\000' > file-without-nulls works beautifully – Paul Carlton Nov 02 '17 at 18:17
  • According to the documentation for GNU sed, \xxx can be used to produce a character whose hexidecimal value is xx. To produce NULL, use \x00. – Alexej Magura Jul 30 '18 at 21:22
9

tr tripped over some other bytes in my file and sed didn't replace anything. I ended up doing it not in sed but in Python:

f = open('file-with-0bytes.dump')
for l in f.readlines():
  print l.replace("\0", '')

Here's a pipeable one-liner:

python -c 'import sys; sys.stdout.write(sys.stdin.read().replace("\0", ""))'

I also noticed some commands actually leave the null bytes there but they're not visible anymore, at least not in an OSX terminal. I used hexdump to debug this.

the
  • 2,819
6

It's quite easy to use Perl to perform a regex. Just replace sed with perl -np -e:

$ printf 'one\0two\0' | perl -np -e 's/\0/\n/g'
one
two

With the -n option, regexes are run line by line, just like sed.

If you want to use zero bytes as record separators, use Perl's -0 option.

$ printf 'one\0two\0' | perl -np0 -e 's/^/prefix /; s/\0/\n/g'
prefix one
prefix two
$ printf 'one\0two\0' | perl -np -e 's/^/prefix /; s/\0/\n/g'
prefix one
two

You can look up the command-line options of Perl by running perldoc perlrun.

Flimm
  • 10,691
  • Can you explain the difference between perldoc perlrun and man perl please? They both look really similar. – cokedude Dec 14 '20 at 19:37
1

To match a null byte, I use this regex with Cygwin's SED:

[^\x01-\x7F]

Big Daddy
  • 11
  • 2
    This is the answer to the question for sed on GNUWin32. It actually strips out more than just nulls. It may or may not work for you depending on what you want to match, and the implementation of sed. – Itsme2003 Apr 27 '18 at 02:54