Importing from an API that has non UTF-8 strings

Question

I am new to both Mathematica and API's. I am trying to access a dictionary of player-ids to names from the website datdota.com, but some of them have chinese characters.

relevant code: URLExecute["http://datdota.com/api/players/101586543"]

When I run this I get the error "cannot import data as JSON format".

I tried to get around it with URLExecute["http://datdota.com/api/players/101586543", CharacterEncoding -> "Unicode"]

but i got the same error.

Is there a different function I need to use? How can I access this data?

Here's a link to what I'm trying to access: http://datdota.com/api/players/101586543

Import["http://datdota.com/api/players/101586543", "JSON"] evaluates fine on my end (11.2 on Linux). — J. M.'s missing motivation, Mar 27 '18 at 03:57
ImportString[FromCharacterCode[URLRead["http://datdota.com/api/players/101586543", "BodyBytes"]], "JSON"] worked on v11.2 Windows. Stems from [https://mathematica.stackexchange.com/questions/154245/who-is-to-blame-parsing-utf8-encoded-json-httpresponse-fails] also URLExecute["http://datdota.com/api/players/101586543", "Text", CharacterEncoding -> "Unicode"] return an encoded result but difficult to return to JSON that can be parsed. — Hans, Mar 27 '18 at 07:12

score 5 · Accepted Answer · edited Mar 28 '18 at 00:57

5

Oh what a mess:

    ImportString[
 FromCharacterCode[
  ToCharacterCode[
   URLExecute["http://datdota.com/api/players/101586543",  "Text"], 
   "UTF8"]], "JSON"]
(*{"data" -> {"nickname" -> "跳刀跳刀丶", "steam_id" -> "101586543"}}*)

V11.2 Windows

In order to answer poster's questions: "Is there a different function I need to use? How can I access this data?" As @J. M. commented

Import["http://datdota.com/api/players/101586543", "JSON"]

should work. Nevertheless, this particular hic-cup you ran into seems to be a known issue in WL when using URLExcecute. The above workaround solution is there just in case you also need to work with API with more parameter passing.

edited Mar 28 '18 at 00:57

J. M.'s missing motivation

124,525
11
401
574

answered Mar 27 '18 at 07:50

Hans

946
6
8

The problem with "Text" may be that it is supposed to infer encoding etc and it is not defined well. "String" should be more stable as kind of raw bytes string but "String" changed in 11.3 while "Text" didn't XD. Daily fun. – Kuba Mar 27 '18 at 08:04
@Kuba "String" works. Also using the encoding "Custom" in the From and To ..Encode seems to work. I tried many other available encodings such as "ShiftJIS", "UTF-8" over "UTF8"some worked, some did not. – Hans Mar 27 '18 at 17:36

Kuba · Answer 2 · 2018-03-27T07:03:07.303

It could be related to all this 'content-type' mess: Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails, but there are no messages issued before failure so I'm not so sure.

I think the issue was known because:

V10.4 returns $Failed and Import::fmterr: Cannot import data as JSON format. >>
V11.2 returns Failure with `ImportFailure tag. No messages.
V11.3 returns correct association

For pre 11.3 use either Import or linked route through "BodyBytes":

ImportString[ FromCharacterCode @ URLRead[ 
  "http://datdota.com/api/players/101586543", "BodyBytes"]
, "RawJSON"
]

<|"data" -> <|"nickname" -> "跳刀跳刀丶", "steam_id" -> "101586543"|>|>

Importing from an API that has non UTF-8 strings

2 Answers2