3

I want to fetch the content of a Persian website. I tried

 URLFetch["http://www.tsetmc.com/"]

Why is Mathematica unable to recode the characters in the results retuned? Why do they look strange? The problem is not a matter of the language because, for some Persian websites, the function works. But what goes wrong when I access this particular website?

Here is an extract from the data URLFetch retuned

:\.18ìî?'û \.0fήøoí]Ùim/*¯^ï\é\[CenterDot]º îô \
\.16û\[CenterDot]ÂÞ¥?Þhî\.04í \[RawEscape]kíîfûB¿»Óܣ͸ÿy?¾\
\[Paragraph]ظq©ñ\.b3þ­¥3Ø©¦u5X®këívðoÿ \.06¯ýd-ýD/7køfðó_ü \
\.02¦û+nÜë/.éþ&ênÇÅ zq!
\.16ú½ø \.06[Ù{+íns# Þ½í/ÿ[m~çg\.0b?½\.b4ð\.b3Þðj\.b3ÝÝ»½¸\.b3\
\[PlusMinus]¿\[CenterDot]t¦¿\.06/ðâv\[CenterDot]Ó/Æ_âx\.16û¿íGí¸\.b3\
J. M.'s missing motivation
  • 124,525
  • 11
  • 401
  • 574
Morry
  • 585
  • 2
  • 11

3 Answers3

11

That website delivers the URL with a content-encoding of gzip. Apparently, URLFetch does not automatically handle decompressing this. You can decode it manually:

 tempFile = URLSave["http://www.tsetmc.com/", Close[OpenTemporary[]]];
 data = Import[tempFile, {"GZIP", "Text"}];
 DeleteFile[tempFile]
Todd Gayley
  • 4,198
  • 20
  • 19
  • How did you find out? – Morry Jul 26 '13 at 19:23
  • When I save the address below I lose some information of the tables!! http://www.tsetmc.com/Loader.aspxParTree=151311&i=22811176775480091 There should be a way to find all the elements of the page! – Morry Jul 26 '13 at 20:50
  • 1
    @Morry I found out by using a handy HTTP sniffer program called Fiddler. This works great with Mathematica if you have your Help/Internet Connectivity/Proxy Settings set for "Use proxy settings from your system or browser". As for losing some info, it's possible that some of what you see when looking at the page in a browser is delivered via JavaScript callbacks after the page loads. Perhaps that site has a more API-friendly way of serving up data. – Todd Gayley Jul 26 '13 at 22:21
  • 1
    How may I recognize it? When I inspect the element of the page that I look for in Chrome I find it in Chrome Developer Tools... How should I proceed in an efficient way in Mathematica? – Morry Jul 26 '13 at 22:26
  • Could you please indicate a specific element or section of the page that is missing from the Mathematica result, but is present in Chrome? – Todd Gayley Jul 27 '13 at 21:31
  • Consider this page: http://www.tsetmc.com/Loader.aspx?ParTree=151311&i=22560050433388046

    Then the table at the right middle of the page, with light blue header, is not in Mathematica input!! Just simply search for one of the numbers of the table in Mathematica input.

     – Morry Jul 29 '13 at 10:23
  • Is it possible to change it in a way to push it use URLFetch function rather than URLSave? It's because when you are using a CDF file it is not possible to write files on the client's computer! – Morry Aug 04 '13 at 13:06
7

My other answer only addressed the issue of decoding the gzipped content. There is another problem, though, which is that this website uses so-called Dynamic HTML to construct the data that you see in a browser window. In other words, not all the data is delivered in the original HTTP result; rather, it is delivered later by JavaScript callbacks from the browser. What you see in the browser is the assembled product of a program that runs in the browser.

One way to get all the data is to programmatically launch a web browser, tell it to load the page, wait for it to finish all the JavaScript calls needed to acquire all the data, and then ask the browser for the final HTML. This is like automating the "View source" feature in some browsers, and it is often referred to as screen scraping.

On Windows, you can do this easily using .NET/Link:

Needs["NETLink`"];
InstallNET[];
ie = CreateCOMObject["InternetExplorer.Application"];
ie@Navigate["http://www.tsetmc.com/Loader.aspx?ParTree=151311&i=22560050433388046"];
Pause[3];
doc = ie@Document;
html = doc@body@innerHTML;
ReleaseCOMObject[ie]

The html variable now holds the full HTML string.

Note that a Pause is required to ensure that the DHTML callbacks have time to complete. I don't know what a proper pause length would be. A motivated programmer could undoubtedly hook into some sort of "page finished" event that is fired by the browser.

Todd Gayley
  • 4,198
  • 20
  • 19
  • If you mention over the top there are some tabs with further information. Is it possible also to grab the data in the tabs?? – Morry Jul 30 '13 at 10:20
  • And to be honest this method even does not include the table that I was talking about!! Since the link is changed, try this link: http://www.tsetmc.com/Loader.aspx?ParTree=151311&i=778253364357513 – Morry Jul 30 '13 at 10:33
  • It works for me. Did you see my remark about the Pause length? Try increasing the value. As for getting the tabs, that might be tricky. I played around a bit but didn't have any success. – Todd Gayley Jul 30 '13 at 16:19
  • You are right... IT IS WORKING!

    Do you have any idea how I can get the tabs?? Maybe some hints as I am not much familiar with JAVA!!

     – Morry Jul 30 '13 at 16:36
  • Here is my failed attempt to simulate a click on the DPS tab. This builds on the code in my answer. The click method has no effect: all = doc@all; dpsTabElement = Do[If[all@item[i]@innerText == "DPS", Return[all@item[i]]], {i, 0, all@length - 1}]; dpsTabElement@click[] – Todd Gayley Aug 03 '13 at 05:34
1

In 10.3, following should be sufficient to get the desired result:

ImportString[URLFetch["http://www.tsetmc.com/"]]
user1473011
  • 86
  • 5