How to export strings as pure ASCII?

Question

The "PrintableASCII" characters set consists of 97 characters, including space (\[RawSpace]) and two control characters: the newline character \n (\[NewLine]) and the carriage return \r (\[RawReturn]), which are the only allowed WhitespaceCharacters in this encoding (highlighted by light blue in the following table):

printableASCIISet = 
  Cases[ToCharacterCode[FromCharacterCode[Range[0, 255]], "PrintableASCII"], k_Integer];

printableASCIISet // Length

Grid[
  Flatten[
   Transpose /@
    Partition[{#, FromCharacterCode[#]} & /@ printableASCIISet, UpTo@20], 1],
  Frame -> {All, {{True, False}}}, Spacings -> 0.2, FrameStyle -> LightGray] /.
 s_String /; StringMatchQ[s, WhitespaceCharacter] :>
  Item[StringTrim[ToString[InputForm@s], "\""], Background -> LightBlue]

97

The "ASCII" encoding allows all control characters and consists of the characters with codes from 0 to 127.

I wish to allow only newlines (\n), carriage returns (\r), spaces () and tabs (\t) as whitespace characters when exporting as ASCII. So the only difference with "PrintableASCII" is that tabs are also allowed, and of course all the special, extended-ASCII and Unicode characters must be converted into their corresponding FullForms.

Here are example strings with expected output in the exported textual file:

1. Input:

   ExportAsASCII["test1.txt", "Here is a string in quotes:\n\"quoted string\""]
   

   Verbatim contents of the created file "test1.txt":

   Here is a string in quotes: "quoted string"

   (note the absence of the outer quotation marks and also that the inner quotes aren't escaped).

2. Input:

   ExportAsASCII["test2.txt", "Lamé \[LongRightArrow] αβ+"]
   

   Verbatim contents of the created file "test2.txt":

   Lam\[EAcute] \[LongRightArrow] \[Alpha]\[Beta]+

   (note the é encoded as \[EAcute], not as \\[EAcute]).

3. Input:

   s = "\!\(\r\n\*SubsuperscriptBox[\(\[Integral]\), \(-Pi\), \(Pi\)]\(g[\r\n\*SuperscriptBox[\(x\), \(2\)] + \r\n\*SuperscriptBox[\(x\), \(3\)] + \r\n\*SuperscriptBox[\(x\), \(6\)]] \[DifferentialD]x\)\)"
   
   ExportAsASCII["test3.txt", s]
   

   Verbatim contents of the created file "test3.txt":

   \!\( \*SubsuperscriptBox[\(\[Integral]\), \(-Pi\), \(Pi\)]\(g[ \*SuperscriptBox[\(x\), \(2\)] + \*SuperscriptBox[\(x\), \(3\)] + \*SuperscriptBox[\(x\), \(6\)]] \[DifferentialD]x\)\)

   (note that the newline characters aren't escaped and present here as actual new lines).

How is it possible to implement ExportAsASCII?

---

P.S. Actually the above-described behavior is (by the essence) what I expected for Exporting as "String" with CharacterEncoding -> "ASCII" according to the Documentation:

Export["test.txt", "Lamé \[LongRightArrow] αβ+", "String", CharacterEncoding -> "ASCII"]

But currently "ASCII" as well as "PrintableASCII" and None seem to be simply ignored by Export. :(

Answer 1

ExportAsASCII

Currently I know only one well-documented and working (with exception for the NULL character \.00 and a few characters from Unicode Private Use Area which normally shouldn't appear in code) method to convert all special and non-ASCII characters in an arbitrary string into "PrintableASCII" programmatically:

toInputForm[s_String] := StringTake[
   ToString[s, InputForm, CharacterEncoding -> "PrintableASCII"], {2, -2}];

In general this method duplicates every backslash which is a part of a special character like \n, \!, \* etc.:

toInputForm["\!\n"]

"\\!\\n"

... and adds two backslashes before every escaping backslash:

toInputForm["\\"]
toInputForm["\""]
toInputForm["\\n"]

"\\\\"

"\\\""

"\\\\n"

AFAIK the only two characters which have to be escaped inside of the Mathematica strings are backslash \ (the escaping character and the integral part of every special character) and double quote " (which indicates the start and the end of the string). This is important since for performing the reverse transformation we have to care about every such character.

Based on these considerations we can write a list of replacement rules which will perform the reverse transformation according to our goals:

unescape[s_String] := 
  StringReplace[
   s, {"\\\\" -> "\\", "\\\"" -> "\"", "\\n" -> "\n", "\\r" -> "\r", "\\t" -> "\t"}];

Now we can implement ExportAsASCII as follows:

ExportAsASCII[fileName_String, s_String] := 
 Export[fileName, unescape[toInputForm[s]], "String"]

... or perhaps more efficiently using WriteString:

ExportAsASCII[fileName_String, s_String] :=
  Module[{str = OpenWrite[fileName, BinaryFormat -> True]},
   WriteString[str, unescape[toInputForm[s]]]; Close[str]];

... or BinaryWrite:

ExportAsASCII[fileName_String, s_String] :=
  Close[BinaryWrite[fileName, unescape[toInputForm[s]]]];

This implementation passes all the tests given in the question. Note however that this implementation has the same defect as Export[..., "String"] for Unicode characters: in the exported file escaped and unescaped versions of the same non-PrintableASCII character can't be distinguished. For example, both from "\!" and "\\!" Export produces a file containing \!. There is no way to avoid such inconsistency entirely since if we will write \! from "\!" and \\! from "\\!", we still have ambiguity for "\\!" and "\\\!" which both will be written as \\!. That's because we must write "\\" as single backslash \ it actually represents.

---

ImportAsASCII

Since non-PrintableASCII characters are presented in FullForm in the exported file, they will be imported already escaped like "\\[Paragraph]", "\\200" or "\\!". We must avoid escaping them again when we replace "\\" with "\\\\" (the reverse of what unescape does). Let us find all characters which may appear after \ in FullForm of a non-PrintableASCII character (excluding \t which I include in my character set):

mySet = Join[ToCharacterCode["\t"], 
   Cases[ToCharacterCode[FromCharacterCode[Range[0, 255]], "PrintableASCII"], k_Integer]];
disallowed = Complement[Range[0, 65535], mySet];
chs = toInputForm[FromCharacterCode[#]] & /@ disallowed;
sel = Select[chs, StringTake[#, 1] === "\\" &];
DeleteDuplicates[StringTake[#, {2}] & /@ sel]

{"0", "b", "f", "[", "1", "2", ":", ")", "!", "@", "%", "^", "&", "*", "(", "_", "+", "/", "`"}

The named characters (like \[Mod2Key]) and input forms \:xxxx (where xxxx is hexadecimal code) and \xxx (where xxx contains only decimals) can be matched by "\\" ~~ RegularExpression["\\[[A-z12]+\\]|:[a-f\\d]{4}|\\d{3}"] (note that this regex is exhaustive but still excessive, and will match also invalid inputs like \[foo] or \999). There also is partially allowed input form \.xx which normally isn't generated by toInputForm, so I do not account for it.

Hence a regular expression matching "\\" which will protect such characters may be written as

temperedBackslash = 
  RegularExpression["\\\\(?![bf()!@%^&*_+`/]|\\[[A-z12]+\\]|:[a-f\\d]{4}|\\d{3})"];

Here is a function which performs the reverse operation:

escape[s_String] := 
  StringReplace[s, {temperedBackslash -> "\\\\", "\"" -> "\\\"", 
                    "\n" -> "\\n", "\r" -> "\\r", "\t" -> "\\t"}];
ImportAsASCII[fileName_String] :=
 ToExpression["&quot;" <> escape[FromCharacterCode[BinaryReadList[fileName]]] <> "&quot;", 
  InputForm]

Consistency check:

string = "\!\(\r\n\*Lamé \[LongRightArrow] αβ+\\\\\\\\\)⊕\[Wolf]\[RuleDelayed]\"";
string === ImportAsASCII[ExportAsASCII["test1.txt", string]]

True