How to prevent ExportString from reencoding diacritics when exporting as JSON

Question

I'm not sure if it's a bug with Mathematica or if I'm missing something, but when trying to ExportString a UTF-8 string with diacritics as JSON, it's messing up the character encoding and producing invalid JSON. Here's an example:

ExportString[{"1" -> "Conexión"}, "JSON"]

It doesn't matter if I add CharacterEncoding -> "Unicode" or whatever, the output always ends up as

{
    "1": "ConexiÃ\.b3n"
}

Has anybody encountered this problem before, or know how to solve it? I'm running Mathematica 10.0.2, in case that matters.

Answer 1

I don't believe that this is a bug. Here's why:

What exactly happens? Export and ExportString will always encode "JSON" as UTF-8. This may be intentional, as UTF-8 is the default encoding for JSON files. From the documentation:

Strings in the Wolfram Language are represented in JSON as UTF-8 strings, escaped as required by the JSON standard.

Strings in Mathematica have sort of a dual function. They can be used to store:

• text, which might contain various unicode characters such as "γő"
• binary data (including the null character), if they only have character codes 0-255.

The Import/Export formats to use for these two different purposes are "Text" and "String", respectively.

The result of ExportString is supposed to be binary data that can be literally written into a file (without any transformations) and produce a correct output.

How to fix it? The question is why you want to ExportString JSON?

Is it to stitch it together with other strings, and then finally export to a file? Then make sure all other strings are also encoded into UTF-8 before stitching, and Export as "String".

Ot is it to display to the user? Then decode the UTF-8 by re-importing (ImportString) this string representing binary data as "Text":

raw = ExportString["Conexión", "JSON"];

ImportString[raw, "Text"]
(* "\"Conexión\"" *)

An inconsistency The "RawJSON" import/export format behaves differently:

ExportString["ő", "RawJSON"]
(* "\"ő\"" *)

This inconsistency is a bit disturbing because it means that we cannot treat the output of JSON and RawJSON the same way. Exporting this result as "String" would give a text file containing the ASCII characters "o''" and not "ő".

Answer 2

Actually this problem is still in Mma 10.2

here is a solution

ExportString["Conexión", "JSON"]

(*
    "ConexiÃ\.b3n"
*)

ExportString["Conexión", "JSON", CharacterEncoding -> "UTF8"]

(*
    "ConexiÃ\.b3n"
*)

FromCharacterCode[

 ToCharacterCode[ExportString["Conexión", "JSON"]], "UTF8"]

(*
    "Conexión"
*)

ExportString["Conexión", "RawJSON"]

(*
    "Conexión"
*)

ExportString["Conexión", "HTMLFragment"]

Here Actually I want to add a Chinese Word but because this problem...

(*
    Conexión
*)

A problem is sometimes we should find two matched encodings

decode[str_, encode_, decode_] := 

 FromCharacterCode[ToCharacterCode[str, encode], decode]

Here is a collection of some problem about encodings.

Answer 3

I recreated MacroB's answer above and confirm this still is a bug in version 12.1.

Export["fromExport.json", {"1" -> "Conexión"}, "JSON"]
Import["fromExport.json", "JSON"]
stringout = ExportString[{"1" -> "Conexión"}, "JSON"];
filestream = OpenWrite["fromWriteString.json"];
WriteString[filestream, stringout];
Close[filestream];
Import["fromWriteString.json", "JSON"]

It is a bug because the jsons show different strings in Visual Studio Code:

Export[]:

{ "1":"Conexión" }

ExportString[]:

{ "1":"ConexiA~\263n" }

I am using ExportString with HTTPRequest and URLRead to send the json to Python's FastAPI. Now that explains why Python's FastAPI prints weirdness.

This workaround does NOT work either.

Update:

Here is a referesher on UTF-8.

And below is an approach by Wolfram Technical Support:

    byteArray=ExportByteArray[data,"JSON"];
    filestream=OpenWrite["output.json",BinaryFormat->True];
    BinaryWrite[filestream,byteArray];
    Close[filestream];
    Import["output.json","JSON"]

Answer 4

This is unfortunately not an answer, but an extended comment that outgrew the comment form. In short, this might be a bug in ExportString.

I am using Mathematica v. 10.1.0 on Windows 7 - 64 bit. Here's what I tried so far.

---

I tried to export your JSON content to a file, using the same format you proposed. This works just fine.

Export["fromExport.json", {"1" -> "Conexión"}, "JSON"]
Import["fromExport.json", "JSON"]

(* Out: {"1" -> "Conexión"} *)

The JSON file generated above also contains the same information:

{
    "1": "Conexión"
}

I tried other strings, and other diacritics: no problems here.

---

Although I have typically found Export and ExportString to behave very similarly, in this case ExportString fails as you showed on my system as well.

It doesn't seem to be a front-end problem either (i.e. the output is correct, but the front end messes up when formatting it for display). I tried to write the generated string out to a file, and it's still messed up:

stringout = ExportString[{"1" -> "Conexión"}, "JSON"];

filestream = OpenWrite["fromWriteString.json"];
WriteString[filestream, stringout];
Close[filestream];

Import["fromWriteString.json", "JSON"]

(* Out: Import::fmterr: Cannot import data as JSON format. *)

As you said, the JSON string generated is invalid. For completeness, on my system the fromWriteString.json file contains:

{
    "1": "ConexiA~\263n"
}

Hope this helps!

Answer 5

Workarounds via "Text" and ByteArray showing interrelations between binary and textual encodings in String highlighted by Szabolcs (see also this authoritative answer by Itai Seggev):

string = ExportString[{"1" -> "Conexión"}, "JSON"]
Export["test.json", string, "String"]
text = ImportString[string, "Text"]
Export["test2.json", text, "Text"]
Import["test.json"]
Import["test2.json"]
% === %%
bytearray = StringToByteArray@text
ImportByteArray[bytearray, "JSON"]

byteArray2 = ExportByteArray[{"1" -> "Conexión"}, "JSON"]
byteArray2 === bytearray
filestream = OpenWrite["fromBinaryWrite.json", BinaryFormat -> True];
BinaryWrite[filestream, bytearray];
Close[filestream];
Import["fromBinaryWrite.json", "JSON"]

(outputs are from version 12.3.1 on Windows 10)