Explicitly control newline behaviour when writing strings with arbitary encoding

Question

On Windows system, Mathematica by default opens stream which will automatically transform any "\n" into "\r\n". But I want pure line feed. In this post, librik suggested using BinaryFormat->True in OpenWrite to explicitly control newline behaviour.

In this way,

tmp = OpenWrite[file, BinaryFormat -> True];
WriteString[tmp, "test", "\n", "test"];
Close[tmp]

will give correctly

in which, "LF" is line feed corresponding to "\n" or 10th ASCII character.

However, if I want write Chinese characters, this is not working. For example

tmp = OpenWrite[file, BinaryFormat -> True, 
   CharacterEncoding -> "UTF8"];
WriteString[tmp, "测试", "\n", "test"];
Close[tmp]

Even though, I set CharacterEncoding -> "UTF8", the result is

The words are gone!

So librik's solution needs some extension. How to control newline for outputing any string?

george2079 · Answer 1 · 2017-12-28T16:51:35.837

The trouble is it seems when you open BinaryFormat->True it ignores the CharacterEncoding directive. This is a hack writing to a ascii scratch file to extract the utf8 binary encoding:

tobytes[c_String] := Module[
  {b, tmp, scratch = "scratch.file"},
  tmp = OpenWrite[scratch, CharacterEncoding -> "UTF8"];
  WriteString[tmp, c]; Close[tmp];
  tmp = OpenRead[scratch, BinaryFormat -> True];
  b = BinaryReadList[tmp, "Byte"]; Close[tmp];
  b]

f = OpenWrite["out.txt", BinaryFormat -> True]
BinaryWrite[f, tobytes["测试"]]
BinaryWrite[f, "\ntext\n"]
Close[f]

Checking output:

f = OpenRead["out.txt", BinaryFormat -> True]
Flatten@ToCharacterCode[BinaryReadList[f, "Character8"]]
Close[f]

{230, 181, 139, 232, 175, 149, 10, 116, 101, 120, 116, 10}

compare basic non binary WriteString (on windows)

f = OpenWrite["out.txt", CharacterEncoding -> "UTF8"]
WriteString[f, "测试\ntext\n"]
Close[f]
f = OpenRead["out.txt", BinaryFormat -> True]
Flatten@ToCharacterCode[BinaryReadList[f, "Character8"]]
Close[f]

{230, 181, 139, 232, 175, 149, 13, 10, 116, 101, 120, 116, 13, 10}

you see its the same except dropping the "13" ascii CR.

Thank you so much george2079. +1 – matheorem Dec 30 '17 at 14:03 — matheorem, Dec 30 '17 at 14:03

matheorem · Answer 2 · 2018-01-02T06:21:04.547

Thanks to george2079. Inspired by george2079's use of BinaryWrite and ToCharacterCode. I found the whole thing can be as simple as

file = "R:\\test.txt";
BinaryWrite[file, ToCharacterCode["测试\ntest", "UTF-8"]];
Close[file]

I found learning some essential character encoding is important for understanding the low level stuff. I recommend The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

In Short, what happens here is that unicode maps all characters which exists on earth into unique numbers(called code point). UTF-8 is an 8-bit(one byte) encoding scheme to encode these unique numbers. ToCharacterCode transforms any string into UTF-8 encoding and results in a list of integers ranges from 0 to 255(one byte range), BinaryWrite write these integers byte by byte into file. Finally, if you open the file with UTF-8 encoding, you see the right content.

Explicitly control newline behaviour when writing strings with arbitary encoding

2 Answers2